GoFAST : Exploitation
V1.0 |
|
V1.1 |
|
V1.2 |
|
Caution
As a reminder, the GoFAST subscription includes the supervision and management of the application layer as a whole. This guide is intended for the IT manager or the client who wants to understand the operation of GoFAST but under no circumstances perform actions without the approval of CEO-Vision
VM 1
Start-up
The components are started either by service or by a script /opt/ceo-vision/startup.sh
. The components run by the script are as follows:
LibreOffice
Alfresco
Bonita
Solr
Services started at boot:
[root@gofast3-integration ~]# systemctl list-unit-files --state=enabled
UNIT FILE STATE
auditd.service enabled
chronyd.service enabled
crond.service enabled
dbus-org.fedoraproject.FirewallD1.service enabled
firewalld.service enabled
getty@.service enabled
httpd.service enabled
irqbalance.service enabled
microcode.service enabled
mysqld.service enabled
ntpdate.service enabled
php-fpm.service enabled
postfix.service enabled
rsyslog.service enabled
slapd.service enabled
sshd.service enabled
sysstat.service enabled
systemd-readahead-collect.service enabled
systemd-readahead-drop.service enabled
systemd-readahead-replay.service enabled
tomcat@alfresco.service enabled
tomcat@bonita.service enabled
tuned.service enabled
rpcbind.socket enabled
default.target enabled
multi-user.target enabled
nfs-client.target enabled
remote-fs.target enabled
28 unit files listed.
Process
The GoFAST platform is a technology package with the following main components:
[root@gofast ~]# htop
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||11.0G/11.6G] Tasks: 94, 386 thr, 100 kthr; 1 running
Swp[|||||||||||||||||||||||||||||||||||||| 1.63G/3.87G] Load average: 1.29 0.94 0.55
Uptime: 123 days(!), 01:52:21
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
1 root 20 0 189M 3016 1476 S 0.0 0.0 1h26:52 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
31997 root 20 0 6642M 947M 76 S 0.7 8.0 1h56:44 ├+ java -server -Xms1G -Xmx1G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+Us
30805 root 20 0 183M 0 0 S 0.0 0.0 0:01.07 ├+ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --p
30797 root 20 0 488M 13692 8040 S 0.0 0.1 1:45.00 ├+ php-fpm: master process (/etc/php-fpm.conf)
29617 davfs2 20 0 184M 6700 460 S 0.0 0.1 11:08.05 ├─ mount.davfs -o rw uid=admin localhost:8080/alfresco/webdav /mnt/alfresco_webdav/
27323 root 20 0 288M 216 204 S 0.0 0.0 0:03.93 ├+ /opt/libreoffice5.3/program/oosplash --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstal
20621 ldap 20 0 1648M 114M 9912 S 13.3 1.0 7h33:29 ├+ /usr/sbin/slapd -u ldap -h ldapi:/// ldaps:/// ldap:///
16905 root 20 0 249M 2524 672 S 0.0 0.0 1:57.46 ├+ /usr/sbin/httpd -DFOREGROUND
16859 mysql 20 0 3820M 1514M 1392 S 15.3 12.8 12h00:35 ├+ /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
15833 tomcat 20 0 9694M 3693M 1708 S 16.0 31.2 3h16:35 ├+ /usr/lib/jvm/jre/bin/java -server -Xss1024K -Xms2G -Xmx4G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+U
14933 tomcat 20 0 7889M 1082M 1868 S 0.7 9.1 1h06:36 ├+ /usr/lib/jvm/jre/bin/java -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX
14567 root 20 0 123M 420 312 S 0.0 0.0 2:23.65 ├+ /usr/sbin/crond -n
10356 chrony 20 0 98M 692 484 S 0.0 0.0 0:22.64 ├─ /usr/sbin/chronyd
6389 newrelic 20 0 16896 4 0 S 0.0 0.0 0:00.00 ├+ /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/newrelic/nrsysmond.pid
2633 root 20 0 89892 168 84 S 0.0 0.0 9:33.71 ├+ /usr/libexec/postfix/master -w
2521 memcached 20 0 626M 256M 108 S 0.0 2.2 9h21:36 ├+ /usr/bin/memcached -u memcached -p 11211 -m 512 -c 1024
1027 root 20 0 540M 536 156 S 0.0 0.0 16:20.33 ├+ /usr/bin/python -Es /usr/sbin/tuned -l -P
1025 root 20 0 103M 340 224 S 0.0 0.0 10:52.29 ├+ /usr/sbin/sshd -D
1012 root 20 0 644M 7612 7016 S 0.0 0.1 3h18:30 ├+ /usr/sbin/rsyslogd -n
741 root 20 0 427M 1380 636 S 0.0 0.0 8:06.11 ├+ /usr/sbin/NetworkManager --no-daemon
726 root 20 0 323M 1732 612 S 0.0 0.0 7:58.52 ├+ /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
712 root 20 0 107M 8 4 S 0.0 0.0 0:00.01 ├─ /sbin/agetty --noclear tty1 linux
703 root 20 0 24324 972 772 S 0.0 0.0 28:54.14 ├─ /usr/lib/systemd/systemd-logind
696 dbus 20 0 98480 836 384 S 0.0 0.0 49:38.94 ├+ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
695 root 20 0 396M 2520 1144 S 0.0 0.0 2h13:27 ├+ /usr/bin/vmtoolsd
693 root 20 0 19320 396 260 S 0.0 0.0 29:34.11 ├─ /usr/sbin/irqbalance --foreground
685 polkitd 20 0 515M 1224 536 S 0.0 0.0 12:32.97 ├+ /usr/lib/polkit-1/polkitd --no-debug
668 root 16 -4 55416 268 160 S 0.0 0.0 9:22.34 ├+ /sbin/auditd -n
539 root 20 0 47372 368 268 S 0.0 0.0 0:02.01 ├─ /usr/lib/systemd/systemd-udevd
533 root 20 0 198M 0 0 S 0.0 0.0 0:00.03 ├─ /usr/sbin/lvmetad -f
511 root 20 0 46612 9692 9288 S 1.3 0.1 6h20:14 └─ /usr/lib/systemd/systemd-journald
2 root 20 0 0 0 0 S 0.0 0.0 0:12.51 kthreadd
30429 root 20 0 0 0 0 D 0.7 0.0 0:01.43 ├─ kworker/u12:0
29534 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/0:1H
29141 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/2:1
29139 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/u12:2
28808 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/1:2
28806 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/0:2
28263 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/3:0
27999 root 20 0 0 0 0 S 0.0 0.0 0:00.16 ├─ kworker/4:0
27003 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ├─ kworker/3:1
Note that all these services are configured to start automatically when the server boots. For this purpose the line /opt/ceo-vision/startup.sh
has been added to the file /etc/rc.local
.
PHP-FPM
PHP-FPM - PHP FastCGI Process Manager is used for communication between a web server and PHP
[root@gofast3-integration ~]# ps aux | grep php-fpm
apache 1767 0.3 1.8 629420 149660 ? S Aug17 31:00 php-fpm: pool www
apache 22586 0.2 1.3 613004 103984 ? S Aug22 6:41 php-fpm: pool www
apache 22614 0.2 1.1 599200 88228 ? S Aug22 6:48 php-fpm: pool www
apache 22615 0.2 1.3 613068 104980 ? S Aug22 6:40 php-fpm: pool www
apache 22627 0.2 1.2 610080 100572 ? S Aug22 6:33 php-fpm: pool www
apache 22628 0.2 1.3 617524 105540 ? S Aug22 6:22 php-fpm: pool www
apache 22629 0.2 1.3 616392 111448 ? S Aug22 6:31 php-fpm: pool www
apache 22630 0.2 1.2 609740 97352 ? S Aug22 6:22 php-fpm: pool www
apache 26294 0.3 1.5 612580 119868 ? S Aug21 15:51 php-fpm: pool www
apache 29923 0.3 2.0 639760 161928 ? S Aug16 39:37 php-fpm: pool www
root 31962 0.0 0.2 499364 20116 ? Ss Aug16 0:51 php-fpm: master process (/etc/php-fpm.conf)
apache 31973 0.3 1.9 629672 157408 ? S Aug16 39:23 php-fpm: pool www
apache 31974 0.3 2.0 637556 162912 ? S Aug16 40:08 php-fpm: pool www
apache 31975 0.3 1.9 633776 157328 ? S Aug16 41:29 php-fpm: pool www
apache 31976 0.3 1.8 635772 149040 ? S Aug16 39:07 php-fpm: pool www
apache 31977 0.3 1.8 631004 148060 ? S Aug16 38:53 php-fpm: pool www
apache 31978 0.3 2.0 633684 160404 ? S Aug16 41:02 php-fpm: pool www
apache 31979 0.3 1.9 628936 154268 ? S Aug16 39:25 php-fpm: pool www
apache 32069 0.3 1.8 627132 150484 ? S Aug16 40:25 php-fpm: pool www
apache 32071 0.3 2.0 639808 161776 ? S Aug16 40:48 php-fpm: pool www
apache 32092 0.3 2.0 637796 163692 ? S Aug16 41:13 php-fpm: pool www
apache 32136 0.3 2.0 637556 165724 ? S Aug16 40:36 php-fpm: pool www
apache 32137 0.3 2.0 641136 161816 ? S Aug16 40:52 php-fpm: pool www
apache 32139 0.3 2.1 647620 171280 ? S Aug16 38:53 php-fpm: pool www
Apache Web Server
In order for the “Portal” part of GoFast, which is based on PHP technology and notably the Drupal CMS, to work, it must be hosted on an Apache server.
In production, many processes are created to respond to each of the http requests made by clients. These processes are named /usr/sbin/httpd
:
[root@gofast3-integration ~]# ps aux | grep httpd
apache 3452 0.0 0.2 581704 16880 ? S 14:14 0:01 /usr/sbin/httpd -DFOREGROUND
apache 4229 0.0 0.2 581740 16820 ? S 14:22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 4261 0.0 0.2 581744 16812 ? S 14:22 0:00 /usr/sbin/httpd -DFOREGROUND
apache 4414 0.0 0.2 581712 16876 ? S 14:24 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5246 0.0 0.2 581668 16512 ? S 14:35 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5247 0.0 0.2 581652 16572 ? S 14:35 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5248 0.0 0.2 581772 16868 ? S 14:35 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5351 0.0 0.2 581740 16620 ? S 14:36 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5462 0.0 0.2 581712 16672 ? S 14:38 0:00 /usr/sbin/httpd -DFOREGROUND
apache 5844 0.0 0.2 581712 16616 ? S 14:44 0:00 /usr/sbin/httpd -DFOREGROUND
root 32021 0.0 0.2 580656 23384 ? Ss Aug16 1:11 /usr/sbin/httpd -DFOREGROUND
Tomcat Web Servers
The “Documentary Warehouse” part is provided by Alfresco software, which is an application developed in Java, which requires a Tomcat web server to operate.
Similarly, the “Process Management” part is provided by Bonitasoft, which is an application developed in Java, which also requires a Tomcat web server to run:
[root@gofast3-integration ~]# ps aux | grep tomcat
tomcat 23103 14.5 30.2 7071740 2410928 ? Ssl Aug22 211:52 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -Djavax.net.ssl.keyStore=/etc/pki/keystore -Djavax.net.ssl.keyStorePassword=a4030ma -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/alfresco -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/alfresco/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
tomcat 23282 0.0 0.0 229540 864 ? Sl Aug22 0:00 /opt/libreoffice5.3/program/oosplash -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat 23297 99.9 0.3 709592 26972 ? Sl Aug22 1458:13 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat 23299 0.0 0.0 0 0 ? Z Aug22 0:00 [soffice.bin] <defunct>
root 29294 0.4 6.2 4298500 495868 ? Sl Aug16 40:58 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http
tomcat 31681 1.1 15.2 6935996 1216136 ? Ssl Aug16 114:10 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -Dbonita.home=/var/bonita -Dsysprop.bonita.db.vendor=mysql -Dbtm.root=/var/lib/tomcats/bonita/ -Dbitronix.tm.configuration=/var/lib/tomcats/bonita/conf/bitronix-config.properties -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/bonita -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/bonita/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
MySQL Database
The two previous components (Drupal and Alfresco) each require a database to enable them to function properly.
These databases are hosted by MySQL.
The database used by Drupal has the name drupal
The database used by Alfresco is called alfresco
In production, this translates into two processes:
/bin/sh /usr/bin/mysqld\_safe –datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql--plugin- dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/va/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
[root@gofast3-integration ~]# ps aux | grep mysqld
mysql 31915 4.5 12.6 2831048 1012492 ? Sl Aug16 447:22 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
Solr search engine
Indexing and searching within the GoFAST platform is provided by Apache Solr.
In production, this results in a process called java –jar start.jar
:
[root@gofast3-integration ~]# ps aux | grep start.jar
root 29294 0.4 6.1 4298500 493744 ? Sl Aug16 41:03 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http
LDAP Server
The various Users and Collaborative Spaces of the GoFAST platform are stored in an LDAP directory, used by the various components of the platform.
In production, this results in a /usr/sbin/slapd
process:
[root@gofast3-integration ~]# ps aux | grep slapd
ldap 30084 0.2 1.1 1235156 95212 ? Ssl Aug07 50:37 /usr/sbin/slapd -u ldap -h ldapi:/// ldaps:/// ldap:///
Document preview component (LibreOffice)
All (compatible) documents stored in the GoFAST platform have a preview in PDF format.
This transformation is carried out by the LibreOffice software.
In production this results in a process named /opt/libreoffice4.1/program/soffice.bin
:
[root@gofast3-integration ~]# ps aux | grep libreoffice
root 23081 0.0 0.0 295076 856 ? Sl Aug22 0:00 /opt/libreoffice5.3/program/oosplash --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser --headless --nocrashreport --nofirststartwizard --nologo --norestore
root 23102 2.7 2.0 1598084 162032 ? Sl Aug22 42:07 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager --headless --nocrashreport --nofirststartwizard --nologo --norestore
tomcat 23282 0.0 0.0 229540 864 ? Sl Aug22 0:00 /opt/libreoffice5.3/program/oosplash -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat 23297 99.9 0.3 709592 26972 ? Sl Aug22 1508:19 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -headless -nocrashreport -nofirststartwizard -nologo -norestore
Notifications by email
The gofast_mail_queue
module is used for email notifications. It implements hook_cron()
to send pending emails.
Cron
Some tasks can be automated using cron (Linux task manager). A cron job is a command defined in the file /etc/crontab
that is executed at a specific time. On VM1, there are the following entries:
* * * * * apache wget -O - -q -t 1 http://localhost/gofast-cron.php
- crongofast-cron.php
runs every minute and handles incoming requests to trigger cron tasks* * * * * root /opt/ceo-vision/restart_soffice.sh
- therestart_soffice.sh
script runs every minute and if it detects a problem with LibreOffice, it restarts it31 23 * * * root /opt/ceo-vision/backup.sh daily
- the backup script runs once a day (at 23:31) and saves the data to/var/backup
31 23 * * root /var/www/d7/scripts/versions-script.sh > /var/www/d7/scripts/versions-script_output.txt
- the script retrieving the versions of the components runs once a day (at 11:31) and fills the fileversions-script_output.txt
with the output of the script
Logs
Log files are very important because when there is a problem they allow you to see all the events of the system. Most logs are located in /var/logs/
. Below are the locations of the logs on the VM1:
/var/log/php-fpm/ |
PHP-FPM |
/etc/httpd/logs/ |
Apache |
/var/lib/tomcats/alfresco/logs/ |
Alfresco |
/var/lib/tomcats/bonita/logs/ |
Bonita |
/var/log/mysqld.log |
MySQL |
/var/log/mysql.log |
|
/var/log/mysql-slow.log |
|
/opt/solr/server/logs/ |
Solr |
/var/log/ldap.log |
LDAP |
- |
LibreOffice |
/var/log/aide/aide.log |
Help |
/var/www/d7/sites/default/files/logs |
Alfresco_script |
/var/log/audit/audit.log |
Audit |
/var/log/boot.log |
Boot |
/var/log/clamd.log |
Clamd |
/var/log/dmesg |
Dmesg |
/var/log/drupal |
Drupal |
/var/www/d7/sites/default/files/logs/error_js.log |
Error_js |
/var/log/firewalld |
Firewalld |
/var/log/freshclam.log |
Freshclam |
/var/log/grubby |
Grubby |
/var/log/grubby_prune_debug |
Grubby_prune_debug |
/var/log/messages |
Logrotate |
/var/log/messages |
Python |
/var/log/messages |
Rsyslogd |
/var/log/secure |
Secure |
/var/log/messages |
Server |
/var/log/messages |
Setroubleshoot |
/var/log/messages |
Sudo |
/var/log/messages |
Systemd |
/var/log/messages |
Systemd-logind |
/var/log/tomcat/catalina.out |
Tomcat |
/var/log/wtmp |
Wtmp |
/var/log/zabbix/zabbix_agentd.log |
Zabbix_agentd |
VM 2
Start-up
Services started at boot:
[centos@gofast-comm-ceov-prod ~]$ systemctl list-unit-files --state=enabled
UNIT FILE STATE
auditd.service enabled
autovt@.service enabled
chronyd.service enabled
cloud-config.service enabled
cloud-final.service enabled
cloud-init-local.service enabled
cloud-init.service enabled
crond.service enabled
dbus-org.fedoraproject.FirewallD1.service enabled
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.nm-dispatcher.service enabled
firewalld.service enabled
getty@.service enabled
irqbalance.service enabled
kdump.service enabled
microcode.service enabled
NetworkManager-dispatcher.service enabled
NetworkManager.service enabled
postfix.service enabled
postgresql.service enabled
rsyslog.service enabled
sshd.service enabled
systemd-readahead-collect.service enabled
systemd-readahead-drop.service enabled
systemd-readahead-replay.service enabled
tuned.service enabled
default.target enabled
multi-user.target enabled
remote-fs.target enabled
runlevel2.target enabled
runlevel3.target enabled
runlevel4.target enabled
32 unit files listed.
Process
The set of all processes that run on VM2 is below:
[centos@gofast-comm-ceov-prod ~]$ ps aux | cat
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 123032 3572 ? Ss Aug16 0:17 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root 337 0.0 0.3 61804 25936 ? Ss Aug16 0:19 /usr/lib/systemd/systemd-journald
postfix 353 0.0 0.0 89260 3948 ? S 14:58 0:00 cleanup -z -t unix -u
postfix 357 0.0 0.0 89216 4512 ? S 14:58 0:00 local -t unix
root 366 0.0 0.0 43132 1832 ? Ss Aug16 0:00 /usr/lib/systemd/systemd-udevd
root 373 0.0 0.0 114560 1752 ? S<sl Aug16 0:08 /sbin/auditd -n
root 406 0.0 0.0 24200 1696 ? Ss Aug16 0:05 /usr/lib/systemd/systemd-logind
dbus 407 0.0 0.0 24536 1808 ? Ss Aug16 0:09 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
postfix 408 0.0 0.0 89116 3912 ? S 14:59 0:00 trivial-rewrite -n rewrite -t unix -u
root 432 0.0 0.3 327140 26688 ? Ssl Aug16 0:16 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root 433 0.0 0.1 332300 12884 ? Ssl Aug16 0:06 /usr/sbin/rsyslogd -n
root 435 0.0 0.0 19180 1204 ? Ss Aug16 0:24 /usr/sbin/irqbalance --foreground
root 446 0.0 0.0 110036 840 tty1 Ss+ Aug16 0:00 /sbin/agetty --noclear tty1 linux
root 488 0.0 0.1 433520 10392 ? Ssl Aug16 0:03 /usr/sbin/NetworkManager --no-daemon
polkitd 586 0.0 0.2 528256 13836 ? Ssl Aug16 0:03 /usr/lib/polkit-1/polkitd --no-debug
root 640 0.0 0.2 110592 15820 ? S Aug16 0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
root 855 0.0 0.0 80424 3568 ? Ss Aug16 0:07 /usr/sbin/sshd -D
root 857 0.0 0.2 553212 18348 ? Ssl Aug16 1:45 /usr/bin/python -Es /usr/sbin/tuned -l -P
postgres 1407 0.0 0.1 232292 9344 ? S Aug16 0:19 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
centos 1649 0.0 0.0 151040 1820 pts/0 R+ 15:23 0:00 ps aux
centos 1650 0.0 0.0 107916 360 pts/0 S+ 15:23 0:00 cat
root 2037 0.0 0.0 89008 2140 ? Ss Aug16 0:03 /usr/libexec/postfix/master -w
postfix 2039 0.0 0.0 89288 4116 ? S Aug16 0:01 qmgr -l -t unix -u
postgres 2065 0.0 0.0 192036 1556 ? Ss Aug16 0:00 postgres: logger process
postgres 2075 0.0 0.0 232392 2584 ? Ss Aug16 0:00 postgres: checkpointer process
postgres 2076 0.0 0.0 232292 1956 ? Ss Aug16 0:06 postgres: writer process
postgres 2077 0.0 0.0 232292 1688 ? Ss Aug16 0:06 postgres: wal writer process
postgres 2078 0.0 0.0 233108 2920 ? Ss Aug16 0:15 postgres: autovacuum launcher process
postgres 2079 0.0 0.0 192168 1724 ? Ss Aug16 0:18 postgres: stats collector process
redis 5425 0.0 0.0 142900 5956 ? Ssl Aug17 9:55 /usr/bin/redis-server 127.0.0.1:6379
rabbitmq 5445 0.1 0.7 1129872 48388 ? Ssl Aug17 18:42 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@gofast-comm-ceov-prod -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 5460 0.0 0.0 33052 780 ? S Aug17 0:06 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 5525 0.0 0.0 33016 724 ? Ss Aug17 0:00 inet_gethost 4
rabbitmq 5526 0.0 0.0 35140 884 ? S Aug17 0:00 inet_gethost 4
root 5547 0.0 0.1 220832 12040 ? Ss Aug17 3:12 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
ejabberd 5590 0.1 1.0 1062796 71112 ? Sl Aug17 11:46 /opt/ejabberd-16.12/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-16.12 -progname /opt/ejabberd-16.12/bin/erl -- -home /home/ejabberd -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -mnesia dir "/opt/ejabberd-16.12/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -smp enable start
root 5594 0.1 2.7 5885804 187380 ? Sl Aug17 14:57 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/lib/native/linux-64 -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/async-http-client-1.9.31.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-4.1.1.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang-2.4.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/share/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.0.jar:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share/jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpasyncclient-4.1.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-nio-4.4.1.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0-20161221.230043-4.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.1.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-server-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-fileaccess-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-lgpl-dependencies-1.0-20150723.002855-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.9-20160331.174040-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-2.9-20151211.202410-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.9-20160505.143533-15.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.9-20150723.181652-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jitsi-videobridge/lib/jna-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simple-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jzlib-1.1.3.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20170104.194736-218.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/usr/share/jitsi-videobridge/lib/netty-3.10.4.Final.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videobridge/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jitsi-videobridge/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jitsi-videobridge/lib/tinder-1.2.3.jar:/usr/share/jitsi-videobridge/lib/weupnp-0.1.4.jar:/usr/share/jitsi-videobridge/lib/xml-apis-1.0.b2.jar:/usr/share/jitsi-videobridge/lib/xmlpull-1.1.3.4a.jar:/usr/share/jitsi-videobridge/lib/xpp3-1.1.4c.jar:/usr/share/jitsi-videobridge/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.videobridge.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --port=5275 --subdomain=videobridge --host=gofast-comm-dev.ceo-vision.com pis=xmpp,rest
root 5598 0.1 1.8 5850836 128644 ? Sl Aug17 15:13 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jicofo/lib/native/linux-64 -cp /usr/share/jicofo/jicofo.jar:/usr/share/jicofo/lib/agafua-syslog-0.4.jar:/usr/share/jicofo/lib/bccontrib-1.0.jar:/usr/share/jicofo/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jicofo/lib/bcprov-jdk15on-1.54.jar:/usr/share/jicofo/lib/commons-codec-1.6.jar:/usr/share/jicofo/lib/commons-lang-2.4.jar:/usr/share/jicofo/lib/commons-lang3-3.1.jar:/usr/share/jicofo/lib/commons-logging-1.2.jar:/usr/share/jicofo/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jicofo/lib/core-2.0.0.jar:/usr/share/jicofo/lib/dnsjava-2.1.7.jar:/usr/share/jicofo/lib/dom4j-1.6.1.jar:/usr/share/jicofo/lib/fmj-1.0-20160121.172939-10.jar:/usr/share/jicofo/lib/guava-15.0.jar:/usr/share/jicofo/lib/httpclient-4.4.jar:/usr/share/jicofo/lib/httpcore-4.4.jar:/usr/share/jicofo/lib/ice4j-1.1-20161005.095034-20.jar:/usr/share/jicofo/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jicofo/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jicofo/lib/javax.servlet-3.0.0.v201112011016.jar:/usr/share/jicofo/lib/javax.servlet-api-3.1.0.jar:/usr/share/jicofo/lib/jcip-annotations-1.0.jar:/usr/share/jicofo/lib/jetty-ajp-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-continuation-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-http-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-io-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-security-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-server-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-servlet-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-util-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jicofo/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jicofo/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jicofo/lib/jitsi-credentialsstorage-2.9-20150723.181643-1.jar:/usr/share/jicofo/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jicofo/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jicofo/lib/jitsi-protocol-2.9-20160823.174711-4.jar:/usr/share/jicofo/lib/jitsi-protocol-jabber-2.9-20161005.180600-21.jar:/usr/share/jicofo/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jicofo/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jicofo/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jicofo/lib/jna-4.1.0.jar:/usr/share/jicofo/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jicofo/lib/json-simple-1.1.1.jar:/usr/share/jicofo/lib/jzlib-1.1.3.jar:/usr/share/jicofo/lib/libidn-1.15.jar:/usr/share/jicofo/lib/libjitsi-1.0-20161020.193142-197.jar:/usr/share/jicofo/lib/log4j-1.2.14.jar:/usr/share/jicofo/lib/orange-extensions-1.3.0.jar:/usr/share/jicofo/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jicofo/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jicofo/lib/org.osgi.core-4.3.1.jar:/usr/share/jicofo/lib/sdes4j-1.1.3.jar:/usr/share/jicofo/lib/sdp-api-1.0.jar:/usr/share/jicofo/lib/slf4j-api-1.7.7.jar:/usr/share/jicofo/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jicofo/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jicofo/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jicofo/lib/tinder-1.2.3.jar:/usr/share/jicofo/lib/weupnp-0.1.4.jar:/usr/share/jicofo/lib/xml-apis-1.0.b2.jar:/usr/share/jicofo/lib/xmlpull-1.1.3.4a.jar:/usr/share/jicofo/lib/xpp3-1.1.4c.jar:/usr/share/jicofo/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.jicofo.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --user_domain=gofast-comm-dev.ceo-vision.com --user_name=admin --user_password=a4030ma --subdomain=focus --host=localhost --port=5348
onlyoff+ 5615 0.0 0.2 966584 19692 ? Sl Aug17 0:03 statsd
onlyoff+ 5616 0.0 0.3 935480 24248 ? Sl Aug17 0:09 node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+ 5617 0.0 0.7 1281184 50248 ? Sl Aug17 0:40 node /var/www/onlyoffice/documentserver/server/DocService/sources/gc.js
onlyoff+ 5619 0.0 0.5 1055880 36272 ? Sl Aug17 0:22 node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
onlyoff+ 5624 0.0 0.5 1057192 39240 ? Sl Aug17 0:20 node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+ 5680 0.0 11.7 2014796 799768 ? Sl Aug17 0:27 /usr/bin/node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+ 5761 0.0 0.7 1275996 50600 ? Sl Aug17 0:07 /usr/bin/node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+ 5771 0.0 0.8 1302016 60088 ? Sl Aug17 0:57 /usr/bin/node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
ejabberd 5814 0.0 0.0 11536 456 ? Ss Aug17 0:00 inet_gethost 4
ejabberd 5815 0.0 0.0 39376 1164 ? S Aug17 0:00 inet_gethost 4
root 15780 0.0 0.0 140900 5044 ? Ss 09:12 0:00 sshd: centos [priv]
centos 15782 0.0 0.0 141028 2316 ? R 09:12 0:00 sshd: centos@pts/0
centos 15783 0.0 0.0 115516 2152 pts/0 Ss 09:12 0:00 -bash
root 26134 0.0 0.0 57228 1288 ? Ss Aug23 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 26135 0.0 0.0 58132 4324 ? S Aug23 0:14 nginx: worker process
nginx 26136 0.0 0.0 58124 3852 ? S Aug23 0:22 nginx: worker process
nginx 26137 0.0 0.0 57408 1844 ? S Aug23 0:00 nginx: cache manager process
root 26218 0.0 0.0 124172 1584 ? Ss Aug23 0:00 /usr/sbin/crond -n
postfix 32197 0.0 0.0 89112 3892 ? S 14:45 0:00 pickup -l -t unix -u
Reverse proxy nginx
Nginx is a web server as well as a reverse proxy. It is a high performance server with low memory consumption. On GoFAST, nginx is also used to tunnel XMPP traffic especially on port 443 avoiding opening ports in firewalls. NGINX has a “master” process (which performs privileged operations) and several “worker” processes.
In production the “master” process is named /usr/sbin/nginx
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep nginx
root 26134 0.0 0.0 57228 1288 ? Ss Aug23 0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx 26135 0.0 0.0 57988 3852 ? S Aug23 0:04 nginx: worker process
nginx 26136 0.0 0.0 58124 3852 ? S Aug23 0:08 nginx: worker process
nginx 26137 0.0 0.0 57408 1844 ? S Aug23 0:00 nginx: cache manager process
RabbitMQ
RabbitMQ is a mail server that uses the AMQP protocol. It gives applications a common platform to send and receive messages:
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep rabbitmq
rabbitmq 5445 0.1 0.7 1129872 48028 ? Ssl Aug17 28:26 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@gofast-comm-ceov-prod -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 5460 0.0 0.0 33052 780 ? S Aug17 0:09 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 5525 0.0 0.0 33016 724 ? Ss Aug17 0:00 inet_gethost 4
rabbitmq 5526 0.0 0.0 35140 884 ? S Aug17 0:00 inet_gethost 4
Collaborative edition
Collaborative online editing is provided by the OnlyOffice office suite. It allows you to work on documents in Fast or Strict mode. OnlyOffice requires a few dependencies including the PostgreSQL database, the Redis database management system, and the Node.js platform.
PostgreSQL results in a /usr/bin/postgres`
process:
[centos@gofast-comm-ceov-prod sbin]$ ps aux |grep postgres
postgres 1407 0.0 0.1 232292 9344 ? S Aug16 0:21 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
postgres 2065 0.0 0.0 192036 1556 ? Ss Aug16 0:00 postgres: logger process
postgres 2075 0.0 0.0 232392 2584 ? Ss Aug16 0:00 postgres: checkpointer process
postgres 2076 0.0 0.0 232292 1956 ? Ss Aug16 0:07 postgres: writer process
postgres 2077 0.0 0.0 232292 1688 ? Ss Aug16 0:07 postgres: wal writer process
postgres 2078 0.0 0.0 233108 2920 ? Ss Aug16 0:17 postgres: autovacuum launcher process
postgres 2079 0.0 0.0 192168 1724 ? Ss Aug16 0:20 postgres: stats collector process
Redis results in a process called /usr/bin/redis-server
:
[centos@gofast-comm-ceov-prod sbin]$ ps aux |grep redis-server
redis 5425 0.0 0.0 142900 5956 ? Ssl Aug17 10:56 /usr/bin/redis-server 127.0.0.1:6379
Node.js results in a process called /usr/bin/node
:
[centos@gofast-comm-ceov-prod sbin]$ ps aux | rep node
onlyoff+ 5616 0.0 0.3 936504 23576 ? Sl Aug17 0:10 node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+ 5617 0.0 0.6 1277088 46180 ? Sl Aug17 0:44 node /var/www/onlyoffice/documentserver/server/DocService/sources/gc.js
onlyoff+ 5619 0.0 0.5 1055880 36276 ? Sl Aug17 0:24 node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
onlyoff+ 5624 0.0 0.5 1057192 38976 ? Sl Aug17 0:22 node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+ 5680 0.0 11.7 2014796 799996 ? Sl Aug17 0:29 /usr/bin/node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+ 5761 0.0 0.7 1275996 50620 ? Sl Aug17 0:08 /usr/bin/node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+ 5771 0.0 0.9 1303040 61424 ? Sl Aug17 1:03 /usr/bin/node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
Instant messaging
The instant messaging is based on the JSXC application - XMPP JavaScript client. It requires an XMPP server to work so ejabberd was chosen. In production, this results in a process called /opt/ejabberd-16.12/bin/beam.smp
:
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep ejabberd
ejabberd 5590 0.1 1.0 1061772 70960 ? Sl Aug17 11:03 /opt/ejabberd-16.12/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-16.12 -progname /opt/ejabberd-16.12/bin/erl -- -home /home/ejabberd -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -mnesia dir "/opt/ejabberd-16.12/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -smp enable start
ejabberd 5814 0.0 0.0 11536 456 ? Ss Aug17 0:00 inet_gethost 4
ejabberd 5815 0.0 0.0 17884 680 ? S Aug17 0:00 inet_gethost 4
Meeting
To provide the videoconferencing functionality in GoFAST, the open source application Jitsi Meet is used. The two components needed for conferences are jicofo (session management between each participant and the videobridge) and jitsi-videobridge (XMPP server component that allows multi-user communication).
Note
This portion consumes CPU and bandwidth in proportion to the number of participants
In production, jitsi-videobridge /usr/share/jitsi-videobridge/jitsi-videobridge.jar
(a java component required by jitsi-meet) results in a process:
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep jitsi-videobridge
root 5594 0.1 2.7 5885804 187380 ? Sl Aug17 16:40 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/lib/native/linux-64 -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/async-http-client-1.9.31.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-4.1.1.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang-2.4.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/share/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.0.jar:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpasyncclient-4.1.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-nio-4.4.1.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0-20161221.230043-4.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.1.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-server-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-fileaccess-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-lgpl-dependencies-1.0-20150723.002855-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.9-20160331.174040-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-2.9-20151211.202410-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.9-20160505.143533-15.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.9-20150723.181652-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jitsi-videobridge/lib/jna-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simple-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jzlib-1.1.3.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20170104.194736-218.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/usr/share/jitsi-videobridge/lib/netty-3.10.4.Final.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videobridge/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jitsi-videobridge/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jitsi-videobridge/lib/tinder-1.2.3.jar:/usr/share/jitsi-videobridge/lib/weupnp-0.1.4.jar:/usr/share/jitsi-videobridge/lib/xml-apis-1.0.b2.jar:/usr/share/jitsi-videobridge/lib/xmlpull-1.1.3.4a.jar:/usr/share/jitsi-videobridge/lib/xpp3-1.1.4c.jar:/usr/share/jitsi-videobridge/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.videobridge.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --port=5275 --subdomain=videobridge --host=gofast-comm-dev.ceo-vision.com pis=xmpp,rest
Jicofo /usr/share/jicofo/jicofo.jar
(a focus-type java component needed for jitsi-meet) results in a process:
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep jicofo
root 5598 0.1 1.8 5850836 128648 ? Sl Aug17 16:54 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jicofo/lib/native/linux-64 -cp /usr/share/jicofo/jicofo.jar:/usr/share/jicofo/lib/agafua-syslog-0.4.jar:/usr/share/jicofo/lib/bccontrib-1.0.jar:/usr/share/jicofo/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jicofo/lib/bcprov-jdk15on-1.54.jar:/usr/share/jicofo/lib/commons-codec-1.6.jar:/usr/share/jicofo/lib/commons-lang-2.4.jar:/usr/share/jicofo/lib/commons-lang3-3.1.jar:/usr/share/jicofo/lib/commons-logging-1.2.jar:/usr/share/jicofo/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jicofo/lib/core-2.0.0.jar:/usr/share/jicofo/lib/dnsjava-2.1.7.jar:/usr/share/jicofo/lib/dom4j-1.6.1.jar:/usr/share/jicofo/lib/fmj-1.0-20160121.172939-10.jar:/usr/share/jicofo/lib/guava-15.0.jar:/usr/share/jicofo/lib/httpclient-4.4.jar:/usr/share/jicofo/lib/httpcore-4.4.jar:/usr/share/jicofo/lib/ice4j-1.1-20161005.095034-20.jar:/usr/share/jicofo/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jicofo/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jicofo/lib/javax.servlet-3.0.0.v201112011016.jar:/usr/share/jicofo/lib/javax.servlet-api-3.1.0.jar:/usr/share/jicofo/lib/jcip-annotations-1.0.jar:/usr/share/jicofo/lib/jetty-ajp-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-continuation-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-http-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-io-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-security-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-server-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-servlet-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-util-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jicofo/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jicofo/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jicofo/lib/jitsi-credentialsstorage-2.9-20150723.181643-1.jar:/usr/share/jicofo/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jicofo/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jicofo/lib/jitsi-protocol-2.9-20160823.174711-4.jar:/usr/share/jicofo/lib/jitsi-protocol-jabber-2.9-20161005.180600-21.jar:/usr/share/jicof /lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jicofo/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jicofo/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jicofo/lib/jna-4.1.0.jar:/usr/share/jicofo/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jicofo/lib/json-simple-1.1.1.jar:/usr/share/jicofo/lib/jzlib-1.1.3.jar:/usr/share/jicofo/lib/libidn-1.15.jar:/usr/share/jicofo/lib/libjitsi-1.0-20161020.193142-197.jar:/usr/share/jicofo/lib/log4j-1.2.14.jar:/usr/share/jicofo/lib/orange-extensions-1.3.0.jar:/usr/share/jicofo/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jicofo/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jicofo/lib/org.osgi.core-4.3.1.jar:/usr/share/jicofo/lib/sdes4j-1.1.3.jar:/usr/share/jicofo/lib/sdp-api-1.0.jar:/usr/share/jicofo/lib/slf4j-api-1.7.7.jar:/usr/share/jicofo/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jicofo/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jicofo/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jicofo/lib/tinder-1.2.3.jar:/usr/share/jicofo/lib/weupnp-0.1.4.jar:/usr/share/jicofo/lib/xml-apis-1.0.b2.jar:/usr/share/jicofo/lib/xmlpull-1.1.3.4a.jar:/usr/share/jicofo/lib/xpp3-1.1.4c.jar:/usr/share/jicofo/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.jicofo.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --user_domain=gofast-comm-dev.ceo-vision.com --user_name=admin --user_password=a4030ma --subdomain=focus --host=localhost --port=5348
Supervisor
Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX operating systems. In production, this results in a process called /usr/bin/supervisord
:
[centos@gofast-comm-ceov-prod ~]$ ps aux | grep supervisord
root 16731 0.0 0.1 220916 12144 ? Ss Sep01 3:15 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
Logs
Below are the log locations on VM2:
/var/log/nginx/ |
nginx |
/var/log/onlyoffice/ |
OnlyOffice |
- |
PostreSQL |
/var/log/redis/redis.log |
Redis |
- |
Node.js |
/var/log/jitsi/jicofo.log |
Jicofo |
/var/log/jitsi/jvb.log |
Jitsi-videobridge |
/var/log/messages |
Network |
/var/log/secure |
Secure |
/var/log/messages |
Sshd |
/var/log/messages |
Su |
/var/log/messages |
Systemd |
/var/log/wtmp |
Wtmp |
/var/log/zabbix/zabbix_agentd.log |
Zabbix_agentd |
/var/log/maillog |
Maillog |
/var/log/messages |
NetworkManager |
/usr/lib/rpm/rpm.log |
Rpm |
/var/log/yum.log |
Yum |
/var/log/messages |
Lvm |
/var/log/messages |
Dracut |
/var/log/messages |
Ma1sd |
/var/log/messages |
Systemd-journald |
/var/log/messages |
Rabbitmqctl |
/var/log/messages |
Rabbitmq-server |
/usr/lib/node_modules/npm/node_modules/socks |
Yarn-error |
/var/log/cron |
Cron |
/var/log/prosody/prosody.log |
Prosody |
/var/log/supervisor/supervisord.log |
Supervisord |
/var/log/messages |
Setsebool |
Supervision
Server Monitoring
On some offers, we automatically install a component to monitor the main server information.
This component is Newrelic (https://newrelic.com).
Note
In 2018, it is possible that we will change supervision providers
The main supervised information is as follows:
CPU Load
Disk IO
Ram Use
Available disk space
Network Usage
In production, this results in two /usr/sbin/nrsysmond`
processes that make queries to the internet every 3 minutes:
[root@gofast3-integration ~]# ps aux | grep newrelic
root 1296 0.0 0.0 190024 1916 ? Ssl Aug16 0:01 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --port /tmp/.newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_docker=true
root 1303 0.0 0.0 504360 2788 ? Sl Aug16 0:34 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --port /tmp/.newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_docker=true -no-pidfile
root 18457 0.0 0.0 112648 976 pts/1 R+ 16:14 0:00 grep --color=auto newrelic
tomcat 23103 15.0 30.2 7071740 2410948 ? Ssl Aug22 228:27 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -Djavax.net.ssl.keyStore=/etc/pki/keystore -Djavax.net.ssl.keyStorePassword=a4030ma -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/alfresco -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/alfresco/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
root 29294 0.4 6.1 4298500 493744 ? Sl Aug16 41:08 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http
tomcat 31681 1.1 15.2 6935996 1217192 ? Ssl Aug16 114:46 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -Dbonita.home=/var/bonita -Dsysprop.bonita.db.vendor=mysql -Dbtm.root=/var/lib/tomcats/bonita/ -Dbitronix.tm.configuration=/var/lib/tomcats/bonita/conf/bitronix-config.properties -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/bonita -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/bonita/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Data security (backup, DR,…)
The GoFAST platform brings together the strategic content of the organization. Data security must be based on an ‘architecture’ layer (RAID+SAN dual or clustering) coupled with a backup strategy.
Caution
Backup is therefore essential, as are restoration tests
The issue of allowable loss must be addressed, as must the time for restoration. This makes it possible to determine a backup strategy.
A) Remote backup of the entire platform:
By VM snapshot
B) Remote data backup only by vacation:
By saving application data
By full replication of data on a remote server (Disaster Recovery)
By saving files only
C) Remote data backup only in near real-time:
ON STUDY: CONTACT US
VM snapshot backup
In this case, the entire virtual machine is backed up.
Caution
It is recommended to take a daily snapshot of the VM that is out of business hours as there is an impact on performance (input/output or I/O). In addition, in order to ensure the integrity of the snapshot, the application may need to ‘freeze’ the VM for a certain period of time, depending on the technologies used.
Note
When CEO-Vision provides hosting (SaaS) to one of its partners, this type of backup is automatically included
Backup of application data
Once a day at the end of the day, all the information needed for the GoFAST platform is saved in /var/backup`
.
Caution
It is strongly recommended that the maintainer mount /var/backup
on a remote storage
To do this, using the Linux cron mechanism, the command /usr/bin/rsnapshotdaily
is executed once a day. This mechanism calls a backup script created by CEO-Vision (/opt/ceo-vision/backup.sh`) which stores the required data in the ``/var/backup
folder
If a retention period is set, it is possible to retrieve data from 1 or more days before in this folder /var/backup
The data saved are as follows:
the MySQL drupal database
the MySQL database alfresco
the ldap directory
the files of the Documentary Warehouse
Drupal sources
APPENDIX I: GoFAST Tree (VM1)
|
Application & Scripts CEO-Vision/GoFAST |
|
Applications |
|
|
|
|
|
|
|
Backup space (openldap,alfresco, bonita, Drupal) |
|
Application data |
|
|
|
|
|
|
|
Configuration files |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Certificates |
APPENDIX II: Disaster Recovery “Minimal”
Note
This is an extension (option) of the GoFAST subscription, covering the upgrade of an additional environment.
In this case of DR Minimal, the principle is to bring back the backups created by GoFAST scripts (see “Backing up Application Data”), in a remote environment called standby.
The standby machine is a GoFAST installation as such. When the production environment is updated, the DR environment is updated by CEO-Vision.
Note
In order to ensure the integrity of Alfresco on the DR, the date of the saved files must match the date of the database snapshot. This is guaranteed by the script delivered with the GoFAST platform
Case 1) Alfresco remote backup is done in /var/backup
Import of the database
Copy /var/backup/…alfresco to /var/alfresco
Loading the LDAP part
**Case 2) The remote backup of Alfresco is done directly in the ** directory /var/alfresco
Import of the database
Loading the LDAP part
APPENDIX III: Remote Duplication of Files
It may be desirable to save a simple copy of the repository files on another server.
Caution
In this case, only the latest version of the files is saved. Metadata or comments are not saved
Network Reader
The first method is to use a backup software on the server to store the backups. This backup software must be able to back up a “network drive” or directly a Webdav server. In order to limit the bandwidth used and the machine resources it is best to make incremental or differential backups.
The ‘network drive’ has the following address:
https://url_de_la_gofast/alfresco/webdav
e.g.: https://gofast.ceo-vision.com/alfresco/webdav
Of course the username must be the user ‘adm’ who is the only user with access to all the documents of the platform.
Deployment of GoFAST Certificates:
Wildcards:
MAIN :
drop the folder containing the certificates to /opt
copy the certificate to /etc/pki/tls/certs/localhost.crt
copy the key to /etc/pki/tls/private/localhost.key
copy the server-chain to /etc/pki/tls/certs/server-chain.crt
COMM :
copy the certificate to /etc/pki/tls/certs/localhost.crt
copy the key to /etc/pki/tls/private/localhost.key
cat /etc/pki/tls/certs/gofast.crt /etc/pki/tls/private/gofast.key > /etc/pki/tls/certs/gofast.pem
Warning
Once done, restart the “nginx” service on the comm and the “httpd” service on the hand.
- MAIN :
systemctl restart httpd
- COMM :
systemctl restart nginx
Test the services (co-editing, co-editing lock, meeting, chat), fix if they no longer work.
Corrections to be made (if services malfunction):
Problem non-automatic connection to Element chat (COMM):
Commands to run on the COMM platform (in ssh) to fix the non-automatic chat connection problem.
cd /opt/gofast-comm
echo "Enter GoFAST MAIN URL (ex: gofast.ceo-vision.com):"
read url_main
java InstallCert $url_main
cp jssecacerts /etc/pki/ca-trust/extracted/java/cacerts
systemctl restart ma1sd
Problem Locks OnlyOffice and Non-Save Document (MAIN):
Commands to run on the MAIN platform (in ssh) to fix the lock problem that remains closed when a OnlyOffice co-edition is released (beware this problem causes that any edits on the document will not be saved)
cd /opt
echo "Enter GoFAST COMM URL (ex: gofast-comm.ceo-vision.com):"
read url_comm
java InstallCert $url_comm
cp jssecacerts /etc/pki/ca-trust/extracted/java/cacerts
echo "Warning la GED va redémarrer"
sleep 2
systemctl restart tomcat@alfresco
Meeting Certificates (COMM) issue:
Commands to run on the COMM platform (in ssh) to fix the problem of video conferencing at least three impossible. (with video stream):
Warning
The DMS will restart as a result of these commands, so make sure no one is working on it.
server_name=`grep -oP "(?<=domain: \').+?(?=\')" /etc/ma1sd/ma1sd.yaml`
echo "secret_key = mot de passe dans /etc/jitsi/jicofo/config "
read secret_key
rm -Rf /var/lib/prosody/*
cd /opt/gofast-comm/update
systemctl stop crond
prosodyctl stop
systemctl stop jibri
systemctl stop jitsi-videobridge
systemctl stop jicofo
#Generate certificates
prosodyctl cert generate auth.${server_name}
Warning
Enter the following information for the “prosodyctl cert generate auth.${server_name}” command:
mv /var/lib/prosody/auth.$server_name.crt /var/lib/prosody/auth.crt
mv /var/lib/prosody/auth.$server_name.key /var/lib/prosody/auth.key
mv /var/lib/prosody/auth.$server_name.cnf /var/lib/prosody/auth.cnf
ln -sf /var/lib/prosody/auth.crt /etc/pki/ca-trust/source/anchors/auth.crt
update-ca-trust extract -f
prosodyctl start
prosodyctl register videobridge auth.$server_name $secret_key
prosodyctl register jibri auth.$server_name $secret_key
prosodyctl register recorder recorder.$server_name $secret_key
prosodyctl register focus auth.$server_name $secret_key
prosodyctl mod_roster_command subscribe focus.$server_name focus@auth.$server_name
systemctl start jibri
systemctl start jitsi-videobridge
sleep 3
systemctl start jicofo
systemctl start crond
CEO-Vision Backup Policy
Machines hosted by Outscale
Customers’ machines hosted at Outscale are saved in the CEO-Vision repository at OVH.
Repo |
OVH |
|
Retention policy |
7 catering points |
|
Planning |
Every day at 8.30 p.m. |
Machines hosted by Scaleway
Customers’ machines hosted at Scaleway are saved in the CEO-Vision repository at OVH.
Repo |
OVH |
|
Retention policy |
7 catering points |
|
Planning |
Every day at 04:00 a.m. |
Machines hosted by OVH
Customers’ machines hosted at OVH are saved in the CEO-Vision repository at Scaleway.
Repo |
SCAL |
|
Retention policy |
7 catering points |
|
Planning |
Every day at 8:30 p.m. |