GoFAST : Exploitation

V1.0

  • Initial version

V1.1

  • Adding a detailed part on backups

V1.2

  • Addition of the Directory Structure (Annex 1)

Caution

As a reminder, the GoFAST subscription includes the supervision and management of the application layer as a whole. This guide is intended for the IT manager or the client who wants to understand the operation of GoFAST but under no circumstances perform actions without the approval of CEO-Vision

VM 1

Start-up

The components are started either by service or by a script /opt/ceo-vision/startup.sh. The components run by the script are as follows:

  • LibreOffice

  • Alfresco

  • Bonita

  • Solr

Services started at boot:

[root@gofast3-integration ~]# systemctl list-unit-files --state=enabled
UNIT FILE                                 STATE
auditd.service                            enabled
chronyd.service                           enabled
crond.service                             enabled
dbus-org.fedoraproject.FirewallD1.service enabled
firewalld.service                         enabled
getty@.service                            enabled
httpd.service                             enabled
irqbalance.service                        enabled
microcode.service                         enabled
mysqld.service                            enabled
ntpdate.service                           enabled
php-fpm.service                           enabled
postfix.service                           enabled
rsyslog.service                           enabled
slapd.service                             enabled
sshd.service                              enabled
sysstat.service                           enabled
systemd-readahead-collect.service         enabled
systemd-readahead-drop.service            enabled
systemd-readahead-replay.service          enabled
tomcat@alfresco.service                   enabled
tomcat@bonita.service                     enabled
tuned.service                             enabled
rpcbind.socket                            enabled
default.target                            enabled
multi-user.target                         enabled
nfs-client.target                         enabled
remote-fs.target                          enabled

28 unit files listed.

Process

The GoFAST platform is a technology package with the following main components:

 [root@gofast ~]# htop
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||11.0G/11.6G] Tasks: 94, 386 thr, 100 kthr; 1 running
  Swp[||||||||||||||||||||||||||||||||||||||                                       1.63G/3.87G] Load average: 1.29 0.94 0.55
                                                                                            Uptime: 123 days(!), 01:52:21
  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    1 root       20   0  189M  3016  1476 S  0.0  0.0  1h26:52 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
31997 root       20   0 6642M  947M    76 S  0.7  8.0  1h56:44 ├+ java -server -Xms1G -Xmx1G -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+Us
30805 root       20   0  183M     0     0 S  0.0  0.0  0:01.07 ├+ /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --p
30797 root       20   0  488M 13692  8040 S  0.0  0.1  1:45.00 ├+ php-fpm: master process (/etc/php-fpm.conf)
29617 davfs2     20   0  184M  6700   460 S  0.0  0.1 11:08.05 ├─ mount.davfs -o rw uid=admin localhost:8080/alfresco/webdav /mnt/alfresco_webdav/
27323 root       20   0  288M   216   204 S  0.0  0.0  0:03.93 ├+ /opt/libreoffice5.3/program/oosplash --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstal
20621 ldap       20   0 1648M  114M  9912 S 13.3  1.0  7h33:29 ├+ /usr/sbin/slapd -u ldap -h ldapi:/// ldaps:/// ldap:///
16905 root       20   0  249M  2524   672 S  0.0  0.0  1:57.46 ├+ /usr/sbin/httpd -DFOREGROUND
16859 mysql      20   0 3820M 1514M  1392 S 15.3 12.8 12h00:35 ├+ /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
15833 tomcat     20   0 9694M 3693M  1708 S 16.0 31.2  3h16:35 ├+ /usr/lib/jvm/jre/bin/java -server -Xss1024K -Xms2G -Xmx4G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+U
14933 tomcat     20   0 7889M 1082M  1868 S  0.7  9.1  1h06:36 ├+ /usr/lib/jvm/jre/bin/java -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX
14567 root       20   0  123M   420   312 S  0.0  0.0  2:23.65 ├+ /usr/sbin/crond -n
10356 chrony     20   0   98M   692   484 S  0.0  0.0  0:22.64 ├─ /usr/sbin/chronyd
 6389 newrelic   20   0 16896     4     0 S  0.0  0.0  0:00.00 ├+ /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/newrelic/nrsysmond.pid
 2633 root       20   0 89892   168    84 S  0.0  0.0  9:33.71 ├+ /usr/libexec/postfix/master -w
 2521 memcached  20   0  626M  256M   108 S  0.0  2.2  9h21:36 ├+ /usr/bin/memcached -u memcached -p 11211 -m 512 -c 1024
 1027 root       20   0  540M   536   156 S  0.0  0.0 16:20.33 ├+ /usr/bin/python -Es /usr/sbin/tuned -l -P
 1025 root       20   0  103M   340   224 S  0.0  0.0 10:52.29 ├+ /usr/sbin/sshd -D
 1012 root       20   0  644M  7612  7016 S  0.0  0.1  3h18:30 ├+ /usr/sbin/rsyslogd -n
  741 root       20   0  427M  1380   636 S  0.0  0.0  8:06.11 ├+ /usr/sbin/NetworkManager --no-daemon
  726 root       20   0  323M  1732   612 S  0.0  0.0  7:58.52 ├+ /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
  712 root       20   0  107M     8     4 S  0.0  0.0  0:00.01 ├─ /sbin/agetty --noclear tty1 linux
  703 root       20   0 24324   972   772 S  0.0  0.0 28:54.14 ├─ /usr/lib/systemd/systemd-logind
  696 dbus       20   0 98480   836   384 S  0.0  0.0 49:38.94 ├+ /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
  695 root       20   0  396M  2520  1144 S  0.0  0.0  2h13:27 ├+ /usr/bin/vmtoolsd
  693 root       20   0 19320   396   260 S  0.0  0.0 29:34.11 ├─ /usr/sbin/irqbalance --foreground
  685 polkitd    20   0  515M  1224   536 S  0.0  0.0 12:32.97 ├+ /usr/lib/polkit-1/polkitd --no-debug
  668 root       16  -4 55416   268   160 S  0.0  0.0  9:22.34 ├+ /sbin/auditd -n
  539 root       20   0 47372   368   268 S  0.0  0.0  0:02.01 ├─ /usr/lib/systemd/systemd-udevd
  533 root       20   0  198M     0     0 S  0.0  0.0  0:00.03 ├─ /usr/sbin/lvmetad -f
  511 root       20   0 46612  9692  9288 S  1.3  0.1  6h20:14 └─ /usr/lib/systemd/systemd-journald
    2 root       20   0     0     0     0 S  0.0  0.0  0:12.51 kthreadd
30429 root       20   0     0     0     0 D  0.7  0.0  0:01.43 ├─ kworker/u12:0
29534 root        0 -20     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/0:1H
29141 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/2:1
29139 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/u12:2
28808 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/1:2
28806 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/0:2
28263 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/3:0
27999 root       20   0     0     0     0 S  0.0  0.0  0:00.16 ├─ kworker/4:0
27003 root       20   0     0     0     0 S  0.0  0.0  0:00.00 ├─ kworker/3:1

Note that all these services are configured to start automatically when the server boots. For this purpose the line /opt/ceo-vision/startup.sh has been added to the file /etc/rc.local.

PHP-FPM

PHP-FPM - PHP FastCGI Process Manager is used for communication between a web server and PHP

[root@gofast3-integration ~]# ps aux | grep php-fpm
apache    1767  0.3  1.8 629420 149660 ?       S    Aug17  31:00 php-fpm: pool www
apache   22586  0.2  1.3 613004 103984 ?       S    Aug22   6:41 php-fpm: pool www
apache   22614  0.2  1.1 599200 88228 ?        S    Aug22   6:48 php-fpm: pool www
apache   22615  0.2  1.3 613068 104980 ?       S    Aug22   6:40 php-fpm: pool www
apache   22627  0.2  1.2 610080 100572 ?       S    Aug22   6:33 php-fpm: pool www
apache   22628  0.2  1.3 617524 105540 ?       S    Aug22   6:22 php-fpm: pool www
apache   22629  0.2  1.3 616392 111448 ?       S    Aug22   6:31 php-fpm: pool www
apache   22630  0.2  1.2 609740 97352 ?        S    Aug22   6:22 php-fpm: pool www
apache   26294  0.3  1.5 612580 119868 ?       S    Aug21  15:51 php-fpm: pool www
apache   29923  0.3  2.0 639760 161928 ?       S    Aug16  39:37 php-fpm: pool www
root     31962  0.0  0.2 499364 20116 ?        Ss   Aug16   0:51 php-fpm: master process (/etc/php-fpm.conf)
apache   31973  0.3  1.9 629672 157408 ?       S    Aug16  39:23 php-fpm: pool www
apache   31974  0.3  2.0 637556 162912 ?       S    Aug16  40:08 php-fpm: pool www
apache   31975  0.3  1.9 633776 157328 ?       S    Aug16  41:29 php-fpm: pool www
apache   31976  0.3  1.8 635772 149040 ?       S    Aug16  39:07 php-fpm: pool www
apache   31977  0.3  1.8 631004 148060 ?       S    Aug16  38:53 php-fpm: pool www
apache   31978  0.3  2.0 633684 160404 ?       S    Aug16  41:02 php-fpm: pool www
apache   31979  0.3  1.9 628936 154268 ?       S    Aug16  39:25 php-fpm: pool www
apache   32069  0.3  1.8 627132 150484 ?       S    Aug16  40:25 php-fpm: pool www
apache   32071  0.3  2.0 639808 161776 ?       S    Aug16  40:48 php-fpm: pool www
apache   32092  0.3  2.0 637796 163692 ?       S    Aug16  41:13 php-fpm: pool www
apache   32136  0.3  2.0 637556 165724 ?       S    Aug16  40:36 php-fpm: pool www
apache   32137  0.3  2.0 641136 161816 ?       S    Aug16  40:52 php-fpm: pool www
apache   32139  0.3  2.1 647620 171280 ?       S    Aug16  38:53 php-fpm: pool www

Apache Web Server

In order for the “Portal” part of GoFast, which is based on PHP technology and notably the Drupal CMS, to work, it must be hosted on an Apache server.

In production, many processes are created to respond to each of the http requests made by clients. These processes are named /usr/sbin/httpd:

[root@gofast3-integration ~]# ps aux | grep httpd
apache    3452  0.0  0.2 581704 16880 ?        S    14:14   0:01 /usr/sbin/httpd -DFOREGROUND
apache    4229  0.0  0.2 581740 16820 ?        S    14:22   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4261  0.0  0.2 581744 16812 ?        S    14:22   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4414  0.0  0.2 581712 16876 ?        S    14:24   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5246  0.0  0.2 581668 16512 ?        S    14:35   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5247  0.0  0.2 581652 16572 ?        S    14:35   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5248  0.0  0.2 581772 16868 ?        S    14:35   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5351  0.0  0.2 581740 16620 ?        S    14:36   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5462  0.0  0.2 581712 16672 ?        S    14:38   0:00 /usr/sbin/httpd -DFOREGROUND
apache    5844  0.0  0.2 581712 16616 ?        S    14:44   0:00 /usr/sbin/httpd -DFOREGROUND
root     32021  0.0  0.2 580656 23384 ?        Ss   Aug16   1:11 /usr/sbin/httpd -DFOREGROUND

Tomcat Web Servers

The “Documentary Warehouse” part is provided by Alfresco software, which is an application developed in Java, which requires a Tomcat web server to operate.

Similarly, the “Process Management” part is provided by Bonitasoft, which is an application developed in Java, which also requires a Tomcat web server to run:

[root@gofast3-integration ~]# ps aux | grep tomcat
tomcat   23103 14.5 30.2 7071740 2410928 ?     Ssl  Aug22 211:52 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -Djavax.net.ssl.keyStore=/etc/pki/keystore -Djavax.net.ssl.keyStorePassword=a4030ma -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/alfresco -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/alfresco/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
tomcat   23282  0.0  0.0 229540   864 ?        Sl   Aug22   0:00 /opt/libreoffice5.3/program/oosplash -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat   23297 99.9  0.3 709592 26972 ?        Sl   Aug22 1458:13 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat   23299  0.0  0.0      0     0 ?        Z    Aug22   0:00 [soffice.bin] <defunct>
root     29294  0.4  6.2 4298500 495868 ?      Sl   Aug16  40:58 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http
tomcat   31681  1.1 15.2 6935996 1216136 ?     Ssl  Aug16 114:10 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -Dbonita.home=/var/bonita -Dsysprop.bonita.db.vendor=mysql -Dbtm.root=/var/lib/tomcats/bonita/ -Dbitronix.tm.configuration=/var/lib/tomcats/bonita/conf/bitronix-config.properties -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/bonita -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/bonita/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

MySQL Database

The two previous components (Drupal and Alfresco) each require a database to enable them to function properly.

These databases are hosted by MySQL.

The database used by Drupal has the name drupal

The database used by Alfresco is called alfresco

In production, this translates into two processes:

/bin/sh /usr/bin/mysqld\_safe –datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql
/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql--plugin- dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/va/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock
[root@gofast3-integration ~]# ps aux | grep mysqld
mysql    31915  4.5 12.6 2831048 1012492 ?     Sl   Aug16 447:22 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Solr search engine

Indexing and searching within the GoFAST platform is provided by Apache Solr.

In production, this results in a process called java –jar start.jar:

[root@gofast3-integration ~]# ps aux | grep start.jar
root     29294  0.4  6.1 4298500 493744 ?      Sl   Aug16  41:03 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http

LDAP Server

The various Users and Collaborative Spaces of the GoFAST platform are stored in an LDAP directory, used by the various components of the platform.

In production, this results in a /usr/sbin/slapd process:

[root@gofast3-integration ~]# ps aux | grep slapd
ldap     30084  0.2  1.1 1235156 95212 ?       Ssl  Aug07  50:37 /usr/sbin/slapd -u ldap -h ldapi:/// ldaps:/// ldap:///

Document preview component (LibreOffice)

All (compatible) documents stored in the GoFAST platform have a preview in PDF format.

This transformation is carried out by the LibreOffice software.

In production this results in a process named /opt/libreoffice4.1/program/soffice.bin:

[root@gofast3-integration ~]# ps aux | grep libreoffice
root     23081  0.0  0.0 295076   856 ?        Sl   Aug22   0:00 /opt/libreoffice5.3/program/oosplash --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser --headless --nocrashreport --nofirststartwizard --nologo --norestore
root     23102  2.7  2.0 1598084 162032 ?      Sl   Aug22  42:07 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser --accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager --headless --nocrashreport --nofirststartwizard --nologo --norestore
tomcat   23282  0.0  0.0 229540   864 ?        Sl   Aug22   0:00 /opt/libreoffice5.3/program/oosplash -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -env:UserInstallation=file:///var/alfresco/alf_data/oouser -headless -nocrashreport -nofirststartwizard -nologo -norestore
tomcat   23297 99.9  0.3 709592 26972 ?        Sl   Aug22 1508:19 /opt/libreoffice5.3/program/soffice.bin -env:UserInstallation=file:///var/alfresco/alf_data/oouser -accept=socket,host=127.0.0.1,port=8100;urp;StarOffice.ServiceManager -headless -nocrashreport -nofirststartwizard -nologo -norestore

Notifications by email

The gofast_mail_queue module is used for email notifications. It implements hook_cron() to send pending emails.

Cron

Some tasks can be automated using cron (Linux task manager). A cron job is a command defined in the file /etc/crontab that is executed at a specific time. On VM1, there are the following entries:

  • * * * * * apache wget -O - -q -t 1 http://localhost/gofast-cron.php - cron gofast-cron.php runs every minute and handles incoming requests to trigger cron tasks

  • * * * * * root /opt/ceo-vision/restart_soffice.sh - the restart_soffice.sh script runs every minute and if it detects a problem with LibreOffice, it restarts it

  • 31 23 * * * root /opt/ceo-vision/backup.sh daily - the backup script runs once a day (at 23:31) and saves the data to /var/backup

  • 31 23 * * root /var/www/d7/scripts/versions-script.sh > /var/www/d7/scripts/versions-script_output.txt - the script retrieving the versions of the components runs once a day (at 11:31) and fills the file versions-script_output.txt with the output of the script

Logs

Log files are very important because when there is a problem they allow you to see all the events of the system. Most logs are located in /var/logs/. Below are the locations of the logs on the VM1:

/var/log/php-fpm/

PHP-FPM

/etc/httpd/logs/

Apache

/var/lib/tomcats/alfresco/logs/

Alfresco

/var/lib/tomcats/bonita/logs/

Bonita

/var/log/mysqld.log

MySQL

/var/log/mysql.log

/var/log/mysql-slow.log

/opt/solr/server/logs/

Solr

/var/log/ldap.log

LDAP

-

LibreOffice

/var/log/aide/aide.log

Help

/var/www/d7/sites/default/files/logs

Alfresco_script

/var/log/audit/audit.log

Audit

/var/log/boot.log

Boot

/var/log/clamd.log

Clamd

/var/log/dmesg

Dmesg

/var/log/drupal

Drupal

/var/www/d7/sites/default/files/logs/error_js.log

Error_js

/var/log/firewalld

Firewalld

/var/log/freshclam.log

Freshclam

/var/log/grubby

Grubby

/var/log/grubby_prune_debug

Grubby_prune_debug

/var/log/messages

Logrotate

/var/log/messages

Python

/var/log/messages

Rsyslogd

/var/log/secure

Secure

/var/log/messages

Server

/var/log/messages

Setroubleshoot

/var/log/messages

Sudo

/var/log/messages

Systemd

/var/log/messages

Systemd-logind

/var/log/tomcat/catalina.out

Tomcat

/var/log/wtmp

Wtmp

/var/log/zabbix/zabbix_agentd.log

Zabbix_agentd

VM 2

Start-up

Services started at boot:

[centos@gofast-comm-ceov-prod ~]$ systemctl list-unit-files --state=enabled
UNIT FILE                                   STATE
auditd.service                              enabled
autovt@.service                             enabled
chronyd.service                             enabled
cloud-config.service                        enabled
cloud-final.service                         enabled
cloud-init-local.service                    enabled
cloud-init.service                          enabled
crond.service                               enabled
dbus-org.fedoraproject.FirewallD1.service   enabled
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.nm-dispatcher.service  enabled
firewalld.service                           enabled
getty@.service                              enabled
irqbalance.service                          enabled
kdump.service                               enabled
microcode.service                           enabled
NetworkManager-dispatcher.service           enabled
NetworkManager.service                      enabled
postfix.service                             enabled
postgresql.service                          enabled
rsyslog.service                             enabled
sshd.service                                enabled
systemd-readahead-collect.service           enabled
systemd-readahead-drop.service              enabled
systemd-readahead-replay.service            enabled
tuned.service                               enabled
default.target                              enabled
multi-user.target                           enabled
remote-fs.target                            enabled
runlevel2.target                            enabled
runlevel3.target                            enabled
runlevel4.target                            enabled

32 unit files listed.

Process

The set of all processes that run on VM2 is below:

[centos@gofast-comm-ceov-prod ~]$ ps aux | cat
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 123032  3572 ?        Ss   Aug16   0:17 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
root       337  0.0  0.3  61804 25936 ?        Ss   Aug16   0:19 /usr/lib/systemd/systemd-journald
postfix    353  0.0  0.0  89260  3948 ?        S    14:58   0:00 cleanup -z -t unix -u
postfix    357  0.0  0.0  89216  4512 ?        S    14:58   0:00 local -t unix
root       366  0.0  0.0  43132  1832 ?        Ss   Aug16   0:00 /usr/lib/systemd/systemd-udevd
root       373  0.0  0.0 114560  1752 ?        S<sl Aug16   0:08 /sbin/auditd -n
root       406  0.0  0.0  24200  1696 ?        Ss   Aug16   0:05 /usr/lib/systemd/systemd-logind
dbus       407  0.0  0.0  24536  1808 ?        Ss   Aug16   0:09 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
postfix    408  0.0  0.0  89116  3912 ?        S    14:59   0:00 trivial-rewrite -n rewrite -t unix -u
root       432  0.0  0.3 327140 26688 ?        Ssl  Aug16   0:16 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
root       433  0.0  0.1 332300 12884 ?        Ssl  Aug16   0:06 /usr/sbin/rsyslogd -n
root       435  0.0  0.0  19180  1204 ?        Ss   Aug16   0:24 /usr/sbin/irqbalance --foreground
root       446  0.0  0.0 110036   840 tty1     Ss+  Aug16   0:00 /sbin/agetty --noclear tty1 linux
root       488  0.0  0.1 433520 10392 ?        Ssl  Aug16   0:03 /usr/sbin/NetworkManager --no-daemon
polkitd    586  0.0  0.2 528256 13836 ?        Ssl  Aug16   0:03 /usr/lib/polkit-1/polkitd --no-debug
root       640  0.0  0.2 110592 15820 ?        S    Aug16   0:00 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/lib/NetworkManager/dhclient-eth0.conf eth0
root       855  0.0  0.0  80424  3568 ?        Ss   Aug16   0:07 /usr/sbin/sshd -D
root       857  0.0  0.2 553212 18348 ?        Ssl  Aug16   1:45 /usr/bin/python -Es /usr/sbin/tuned -l -P
postgres  1407  0.0  0.1 232292  9344 ?        S    Aug16   0:19 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
centos    1649  0.0  0.0 151040  1820 pts/0    R+   15:23   0:00 ps aux
centos    1650  0.0  0.0 107916   360 pts/0    S+   15:23   0:00 cat
root      2037  0.0  0.0  89008  2140 ?        Ss   Aug16   0:03 /usr/libexec/postfix/master -w
postfix   2039  0.0  0.0  89288  4116 ?        S    Aug16   0:01 qmgr -l -t unix -u
postgres  2065  0.0  0.0 192036  1556 ?        Ss   Aug16   0:00 postgres: logger process
postgres  2075  0.0  0.0 232392  2584 ?        Ss   Aug16   0:00 postgres: checkpointer process
postgres  2076  0.0  0.0 232292  1956 ?        Ss   Aug16   0:06 postgres: writer process
postgres  2077  0.0  0.0 232292  1688 ?        Ss   Aug16   0:06 postgres: wal writer process
postgres  2078  0.0  0.0 233108  2920 ?        Ss   Aug16   0:15 postgres: autovacuum launcher process
postgres  2079  0.0  0.0 192168  1724 ?        Ss   Aug16   0:18 postgres: stats collector process
redis     5425  0.0  0.0 142900  5956 ?        Ssl  Aug17   9:55 /usr/bin/redis-server 127.0.0.1:6379
rabbitmq  5445  0.1  0.7 1129872 48388 ?       Ssl  Aug17  18:42 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@gofast-comm-ceov-prod -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq  5460  0.0  0.0  33052   780 ?        S    Aug17   0:06 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq  5525  0.0  0.0  33016   724 ?        Ss   Aug17   0:00 inet_gethost 4
rabbitmq  5526  0.0  0.0  35140   884 ?        S    Aug17   0:00 inet_gethost 4
root      5547  0.0  0.1 220832 12040 ?        Ss   Aug17   3:12 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
ejabberd  5590  0.1  1.0 1062796 71112 ?       Sl   Aug17  11:46 /opt/ejabberd-16.12/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-16.12 -progname /opt/ejabberd-16.12/bin/erl -- -home /home/ejabberd -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -mnesia dir "/opt/ejabberd-16.12/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -smp enable start
root      5594  0.1  2.7 5885804 187380 ?      Sl   Aug17  14:57 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/lib/native/linux-64 -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/async-http-client-1.9.31.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-4.1.1.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang-2.4.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/share/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.0.jar:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share/jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpasyncclient-4.1.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-nio-4.4.1.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0-20161221.230043-4.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.1.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-server-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-fileaccess-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-lgpl-dependencies-1.0-20150723.002855-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.9-20160331.174040-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-2.9-20151211.202410-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.9-20160505.143533-15.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.9-20150723.181652-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jitsi-videobridge/lib/jna-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simple-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jzlib-1.1.3.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20170104.194736-218.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/usr/share/jitsi-videobridge/lib/netty-3.10.4.Final.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videobridge/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jitsi-videobridge/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jitsi-videobridge/lib/tinder-1.2.3.jar:/usr/share/jitsi-videobridge/lib/weupnp-0.1.4.jar:/usr/share/jitsi-videobridge/lib/xml-apis-1.0.b2.jar:/usr/share/jitsi-videobridge/lib/xmlpull-1.1.3.4a.jar:/usr/share/jitsi-videobridge/lib/xpp3-1.1.4c.jar:/usr/share/jitsi-videobridge/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.videobridge.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --port=5275 --subdomain=videobridge --host=gofast-comm-dev.ceo-vision.com pis=xmpp,rest
root      5598  0.1  1.8 5850836 128644 ?      Sl   Aug17  15:13 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jicofo/lib/native/linux-64 -cp /usr/share/jicofo/jicofo.jar:/usr/share/jicofo/lib/agafua-syslog-0.4.jar:/usr/share/jicofo/lib/bccontrib-1.0.jar:/usr/share/jicofo/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jicofo/lib/bcprov-jdk15on-1.54.jar:/usr/share/jicofo/lib/commons-codec-1.6.jar:/usr/share/jicofo/lib/commons-lang-2.4.jar:/usr/share/jicofo/lib/commons-lang3-3.1.jar:/usr/share/jicofo/lib/commons-logging-1.2.jar:/usr/share/jicofo/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jicofo/lib/core-2.0.0.jar:/usr/share/jicofo/lib/dnsjava-2.1.7.jar:/usr/share/jicofo/lib/dom4j-1.6.1.jar:/usr/share/jicofo/lib/fmj-1.0-20160121.172939-10.jar:/usr/share/jicofo/lib/guava-15.0.jar:/usr/share/jicofo/lib/httpclient-4.4.jar:/usr/share/jicofo/lib/httpcore-4.4.jar:/usr/share/jicofo/lib/ice4j-1.1-20161005.095034-20.jar:/usr/share/jicofo/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jicofo/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jicofo/lib/javax.servlet-3.0.0.v201112011016.jar:/usr/share/jicofo/lib/javax.servlet-api-3.1.0.jar:/usr/share/jicofo/lib/jcip-annotations-1.0.jar:/usr/share/jicofo/lib/jetty-ajp-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-continuation-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-http-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-io-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-security-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-server-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-servlet-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-util-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jicofo/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jicofo/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jicofo/lib/jitsi-credentialsstorage-2.9-20150723.181643-1.jar:/usr/share/jicofo/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jicofo/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jicofo/lib/jitsi-protocol-2.9-20160823.174711-4.jar:/usr/share/jicofo/lib/jitsi-protocol-jabber-2.9-20161005.180600-21.jar:/usr/share/jicofo/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jicofo/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jicofo/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jicofo/lib/jna-4.1.0.jar:/usr/share/jicofo/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jicofo/lib/json-simple-1.1.1.jar:/usr/share/jicofo/lib/jzlib-1.1.3.jar:/usr/share/jicofo/lib/libidn-1.15.jar:/usr/share/jicofo/lib/libjitsi-1.0-20161020.193142-197.jar:/usr/share/jicofo/lib/log4j-1.2.14.jar:/usr/share/jicofo/lib/orange-extensions-1.3.0.jar:/usr/share/jicofo/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jicofo/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jicofo/lib/org.osgi.core-4.3.1.jar:/usr/share/jicofo/lib/sdes4j-1.1.3.jar:/usr/share/jicofo/lib/sdp-api-1.0.jar:/usr/share/jicofo/lib/slf4j-api-1.7.7.jar:/usr/share/jicofo/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jicofo/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jicofo/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jicofo/lib/tinder-1.2.3.jar:/usr/share/jicofo/lib/weupnp-0.1.4.jar:/usr/share/jicofo/lib/xml-apis-1.0.b2.jar:/usr/share/jicofo/lib/xmlpull-1.1.3.4a.jar:/usr/share/jicofo/lib/xpp3-1.1.4c.jar:/usr/share/jicofo/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.jicofo.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --user_domain=gofast-comm-dev.ceo-vision.com --user_name=admin --user_password=a4030ma --subdomain=focus --host=localhost --port=5348
onlyoff+  5615  0.0  0.2 966584 19692 ?        Sl   Aug17   0:03 statsd
onlyoff+  5616  0.0  0.3 935480 24248 ?        Sl   Aug17   0:09 node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+  5617  0.0  0.7 1281184 50248 ?       Sl   Aug17   0:40 node /var/www/onlyoffice/documentserver/server/DocService/sources/gc.js
onlyoff+  5619  0.0  0.5 1055880 36272 ?       Sl   Aug17   0:22 node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
onlyoff+  5624  0.0  0.5 1057192 39240 ?       Sl   Aug17   0:20 node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+  5680  0.0 11.7 2014796 799768 ?      Sl   Aug17   0:27 /usr/bin/node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+  5761  0.0  0.7 1275996 50600 ?       Sl   Aug17   0:07 /usr/bin/node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+  5771  0.0  0.8 1302016 60088 ?       Sl   Aug17   0:57 /usr/bin/node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
ejabberd  5814  0.0  0.0  11536   456 ?        Ss   Aug17   0:00 inet_gethost 4
ejabberd  5815  0.0  0.0  39376  1164 ?        S    Aug17   0:00 inet_gethost 4
root     15780  0.0  0.0 140900  5044 ?        Ss   09:12   0:00 sshd: centos [priv]
centos   15782  0.0  0.0 141028  2316 ?        R    09:12   0:00 sshd: centos@pts/0
centos   15783  0.0  0.0 115516  2152 pts/0    Ss   09:12   0:00 -bash
root     26134  0.0  0.0  57228  1288 ?        Ss   Aug23   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    26135  0.0  0.0  58132  4324 ?        S    Aug23   0:14 nginx: worker process
nginx    26136  0.0  0.0  58124  3852 ?        S    Aug23   0:22 nginx: worker process
nginx    26137  0.0  0.0  57408  1844 ?        S    Aug23   0:00 nginx: cache manager process
root     26218  0.0  0.0 124172  1584 ?        Ss   Aug23   0:00 /usr/sbin/crond -n
postfix  32197  0.0  0.0  89112  3892 ?        S    14:45   0:00 pickup -l -t unix -u

Reverse proxy nginx

Nginx is a web server as well as a reverse proxy. It is a high performance server with low memory consumption. On GoFAST, nginx is also used to tunnel XMPP traffic especially on port 443 avoiding opening ports in firewalls. NGINX has a “master” process (which performs privileged operations) and several “worker” processes.

In production the “master” process is named /usr/sbin/nginx

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep nginx
root     26134  0.0  0.0  57228  1288 ?        Ss   Aug23   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx    26135  0.0  0.0  57988  3852 ?        S    Aug23   0:04 nginx: worker process
nginx    26136  0.0  0.0  58124  3852 ?        S    Aug23   0:08 nginx: worker process
nginx    26137  0.0  0.0  57408  1844 ?        S    Aug23   0:00 nginx: cache manager process

RabbitMQ

RabbitMQ is a mail server that uses the AMQP protocol. It gives applications a common platform to send and receive messages:

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep rabbitmq
rabbitmq  5445  0.1  0.7 1129872 48028 ?       Ssl  Aug17  28:26 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@gofast-comm-ceov-prod -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@gofast-comm-ceov-prod-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@gofast-comm-ceov-prod" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq  5460  0.0  0.0  33052   780 ?        S    Aug17   0:09 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq  5525  0.0  0.0  33016   724 ?        Ss   Aug17   0:00 inet_gethost 4
rabbitmq  5526  0.0  0.0  35140   884 ?        S    Aug17   0:00 inet_gethost 4

Collaborative edition

Collaborative online editing is provided by the OnlyOffice office suite. It allows you to work on documents in Fast or Strict mode. OnlyOffice requires a few dependencies including the PostgreSQL database, the Redis database management system, and the Node.js platform.

PostgreSQL results in a /usr/bin/postgres` process:

[centos@gofast-comm-ceov-prod sbin]$ ps aux |grep postgres
postgres  1407  0.0  0.1 232292  9344 ?        S    Aug16   0:21 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
postgres  2065  0.0  0.0 192036  1556 ?        Ss   Aug16   0:00 postgres: logger process
postgres  2075  0.0  0.0 232392  2584 ?        Ss   Aug16   0:00 postgres: checkpointer process
postgres  2076  0.0  0.0 232292  1956 ?        Ss   Aug16   0:07 postgres: writer process
postgres  2077  0.0  0.0 232292  1688 ?        Ss   Aug16   0:07 postgres: wal writer process
postgres  2078  0.0  0.0 233108  2920 ?        Ss   Aug16   0:17 postgres: autovacuum launcher process
postgres  2079  0.0  0.0 192168  1724 ?        Ss   Aug16   0:20 postgres: stats collector process

Redis results in a process called /usr/bin/redis-server:

[centos@gofast-comm-ceov-prod sbin]$ ps aux |grep redis-server
redis     5425  0.0  0.0 142900  5956 ?        Ssl  Aug17  10:56 /usr/bin/redis-server 127.0.0.1:6379

Node.js results in a process called /usr/bin/node:

[centos@gofast-comm-ceov-prod sbin]$ ps aux | rep node
onlyoff+  5616  0.0  0.3 936504 23576 ?        Sl   Aug17   0:10 node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+  5617  0.0  0.6 1277088 46180 ?       Sl   Aug17   0:44 node /var/www/onlyoffice/documentserver/server/DocService/sources/gc.js
onlyoff+  5619  0.0  0.5 1055880 36276 ?       Sl   Aug17   0:24 node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js
onlyoff+  5624  0.0  0.5 1057192 38976 ?       Sl   Aug17   0:22 node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+  5680  0.0 11.7 2014796 799996 ?      Sl   Aug17   0:29 /usr/bin/node /var/www/onlyoffice/documentserver/server/SpellChecker/sources/server.js
onlyoff+  5761  0.0  0.7 1275996 50620 ?       Sl   Aug17   0:08 /usr/bin/node /var/www/onlyoffice/documentserver/server/FileConverter/sources/convertermaster.js
onlyoff+  5771  0.0  0.9 1303040 61424 ?       Sl   Aug17   1:03 /usr/bin/node /var/www/onlyoffice/documentserver/server/DocService/sources/server.js

Instant messaging

The instant messaging is based on the JSXC application - XMPP JavaScript client. It requires an XMPP server to work so ejabberd was chosen. In production, this results in a process called /opt/ejabberd-16.12/bin/beam.smp:

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep ejabberd
ejabberd  5590  0.1  1.0 1061772 70960 ?       Sl   Aug17  11:03 /opt/ejabberd-16.12/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-16.12 -progname /opt/ejabberd-16.12/bin/erl -- -home /home/ejabberd -- -sname ejabberd@localhost -noshell -noinput -noshell -noinput -mnesia dir "/opt/ejabberd-16.12/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -smp enable start
ejabberd  5814  0.0  0.0  11536   456 ?        Ss   Aug17   0:00 inet_gethost 4
ejabberd  5815  0.0  0.0  17884   680 ?        S    Aug17   0:00 inet_gethost 4

Meeting

To provide the videoconferencing functionality in GoFAST, the open source application Jitsi Meet is used. The two components needed for conferences are jicofo (session management between each participant and the videobridge) and jitsi-videobridge (XMPP server component that allows multi-user communication).

Note

This portion consumes CPU and bandwidth in proportion to the number of participants

In production, jitsi-videobridge /usr/share/jitsi-videobridge/jitsi-videobridge.jar (a java component required by jitsi-meet) results in a process:

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep jitsi-videobridge
root      5594  0.1  2.7 5885804 187380 ?      Sl   Aug17  16:40 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jitsi-videobridge/lib/native/linux-64 -cp /usr/share/jitsi-videobridge/jitsi-videobridge.jar:/usr/share/jitsi-videobridge/lib/agafua-syslog-0.4.jar:/usr/share/jitsi-videobridge/lib/async-http-client-1.9.31.jar:/usr/share/jitsi-videobridge/lib/bccontrib-1.0.jar:/usr/share/jitsi-videobridge/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/bcprov-jdk15on-1.54.jar:/usr/share/jitsi-videobridge/lib/callstats-java-sdk-4.1.1.jar:/usr/share/jitsi-videobridge/lib/commons-codec-1.6.jar:/usr/share/jitsi-videobridge/lib/commons-lang-2.4.jar:/usr/share/jitsi-videobridge/lib/commons-lang3-3.1.jar:/usr/share/jitsi-videobridge/lib/commons-logging-1.2.jar:/usr/share/jitsi-videobridge/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jitsi-videobridge/lib/core-2.0.0.jar:/usr/share/jitsi-videobridge/lib/dnsjava-2.1.7.jar:/usr/share/jitsi-videobridge/lib/dom4j-1.6.1.jar:/usr/share jitsi-videobridge/lib/fmj-1.0-20161207.221530-23.jar:/usr/share/jitsi-videobridge/lib/gson-2.3.1.jar:/usr/share/jitsi-videobridge/lib/guava-15.0.jar:/usr/share/jitsi-videobridge/lib/httpasyncclient-4.1.jar:/usr/share/jitsi-videobridge/lib/httpclient-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-4.4.jar:/usr/share/jitsi-videobridge/lib/httpcore-nio-4.4.1.jar:/usr/share/jitsi-videobridge/lib/ice4j-2.0-20161221.230043-4.jar:/usr/share/jitsi-videobridge/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jitsi-videobridge/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jitsi-videobridge/lib/javax.servlet-api-3.1.0.jar:/usr/share/jitsi-videobridge/lib/jcip-annotations-1.0.jar:/usr/share/jitsi-videobridge/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-http-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-io-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-rewrite-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-security-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-server-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-servlet-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-util-9.2.10.v20150310.jar:/usr/share/jitsi-videobridge/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jitsi-videobridge/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jitsi-videobridge/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-fileaccess-2.9-20150723.181644-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-lgpl-dependencies-1.0-20150723.002855-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-packetlogging-2.9-20160331.174040-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-2.9-20151211.202410-3.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-jabber-2.9-20160505.143533-15.jar:/usr/share/jitsi-videobridge/lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-resourcemanager-2.9-20150723.181652-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jitsi-videobridge/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jitsi-videobridge/lib/jna-4.1.0.jar:/usr/share/jitsi-videobridge/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jitsi-videobridge/lib/jose4j-0.5.1.jar:/usr/share/jitsi-videobridge/lib/json-simple-1.1.1.jar:/usr/share/jitsi-videobridge/lib/jul-to-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/jzlib-1.1.3.jar:/usr/share/jitsi-videobridge/lib/libidn-1.15.jar:/usr/share/jitsi-videobridge/lib/libjitsi-1.0-20170104.194736-218.jar:/usr/share/jitsi-videobridge/lib/log4j-1.2.15.jar:/usr/share/jitsi-videobridge/lib/log4j-api-2.3.jar:/usr/share/jitsi-videobridge/lib/log4j-core-2.3.jar:/usr/share/jitsi-videobridge/lib/netty-3.10.4.Final.jar:/usr/share/jitsi-videobridge/lib/orange-extensions-1.3.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jitsi-videobridge/lib/org.osgi.core-4.3.1.jar:/usr/share/jitsi-videobridge/lib/osgi-over-slf4j-1.7.7.jar:/usr/share/jitsi-videobridge/lib/sdes4j-1.1.3.jar:/usr/share/jitsi-videobridge/lib/sdp-api-1.0.jar:/usr/share/jitsi-videobridge/lib/sigar-1.6.4.jar:/usr/share/jitsi-videobridge/lib/slf4j-api-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jitsi-videobridge/lib/slf4j-simple-1.6.1.jar:/usr/share/jitsi-videobridge/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jitsi-videobridge/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jitsi-videobridge/lib/tinder-1.2.3.jar:/usr/share/jitsi-videobridge/lib/weupnp-0.1.4.jar:/usr/share/jitsi-videobridge/lib/xml-apis-1.0.b2.jar:/usr/share/jitsi-videobridge/lib/xmlpull-1.1.3.4a.jar:/usr/share/jitsi-videobridge/lib/xpp3-1.1.4c.jar:/usr/share/jitsi-videobridge/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.videobridge.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --port=5275 --subdomain=videobridge --host=gofast-comm-dev.ceo-vision.com pis=xmpp,rest

Jicofo /usr/share/jicofo/jicofo.jar (a focus-type java component needed for jitsi-meet) results in a process:

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep jicofo
root      5598  0.1  1.8 5850836 128648 ?      Sl   Aug17  16:54 java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djava.library.path=/usr/share/jicofo/lib/native/linux-64 -cp /usr/share/jicofo/jicofo.jar:/usr/share/jicofo/lib/agafua-syslog-0.4.jar:/usr/share/jicofo/lib/bccontrib-1.0.jar:/usr/share/jicofo/lib/bcpkix-jdk15on-1.54.jar:/usr/share/jicofo/lib/bcprov-jdk15on-1.54.jar:/usr/share/jicofo/lib/commons-codec-1.6.jar:/usr/share/jicofo/lib/commons-lang-2.4.jar:/usr/share/jicofo/lib/commons-lang3-3.1.jar:/usr/share/jicofo/lib/commons-logging-1.2.jar:/usr/share/jicofo/lib/concurrentlinkedhashmap-lru-1.0_jdk5.jar:/usr/share/jicofo/lib/core-2.0.0.jar:/usr/share/jicofo/lib/dnsjava-2.1.7.jar:/usr/share/jicofo/lib/dom4j-1.6.1.jar:/usr/share/jicofo/lib/fmj-1.0-20160121.172939-10.jar:/usr/share/jicofo/lib/guava-15.0.jar:/usr/share/jicofo/lib/httpclient-4.4.jar:/usr/share/jicofo/lib/httpcore-4.4.jar:/usr/share/jicofo/lib/ice4j-1.1-20161005.095034-20.jar:/usr/share/jicofo/lib/jain-sip-ri-ossonly-1.2.98c7f8c-jitsi-oss1.jar:/usr/share/jicofo/lib/java-sdp-nist-bridge-1.1.jar:/usr/share/jicofo/lib/javax.servlet-3.0.0.v201112011016.jar:/usr/share/jicofo/lib/javax.servlet-api-3.1.0.jar:/usr/share/jicofo/lib/jcip-annotations-1.0.jar:/usr/share/jicofo/lib/jetty-ajp-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-client-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-continuation-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-http-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-io-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-proxy-9.2.10.v20150310.jar:/usr/share/jicofo/lib/jetty-security-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-server-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-servlet-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-util-8.1.16.v20140903.jar:/usr/share/jicofo/lib/jetty-webapp-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jetty-xml-7.0.1.v20091125.jar:/usr/share/jicofo/lib/jicoco-1.0-20161122.205032-37.jar:/usr/share/jicofo/lib/jitsi-android-osgi-1.0-20150723.002702-1.jar:/usr/share/jicofo/lib/jitsi-configuration-2.9-20150723.181638-1.jar:/usr/share/jicofo/lib/jitsi-credentialsstorage-2.9-20150723.181643-1.jar:/usr/share/jicofo/lib/jitsi-dnsservice-2.9-20150723.181644-1.jar:/usr/share/jicofo/lib/jitsi-netaddr-2.9-20150723.181645-1.jar:/usr/share/jicofo/lib/jitsi-protocol-2.9-20160823.174711-4.jar:/usr/share/jicofo/lib/jitsi-protocol-jabber-2.9-20161005.180600-21.jar:/usr/share/jicof /lib/jitsi-protocol-media-2.9-20150723.181646-1.jar:/usr/share/jicofo/lib/jitsi-ui-service-2.9-20150723.181654-1.jar:/usr/share/jicofo/lib/jitsi-util-2.9-20160203.194039-2.jar:/usr/share/jicofo/lib/jna-4.1.0.jar:/usr/share/jicofo/lib/jnsapi-0.0.3-jitsi-1-20151013.145326-2.jar:/usr/share/jicofo/lib/json-simple-1.1.1.jar:/usr/share/jicofo/lib/jzlib-1.1.3.jar:/usr/share/jicofo/lib/libidn-1.15.jar:/usr/share/jicofo/lib/libjitsi-1.0-20161020.193142-197.jar:/usr/share/jicofo/lib/log4j-1.2.14.jar:/usr/share/jicofo/lib/orange-extensions-1.3.0.jar:/usr/share/jicofo/lib/org.apache.felix.framework-4.4.0.jar:/usr/share/jicofo/lib/org.apache.felix.main-4.4.0.jar:/usr/share/jicofo/lib/org.osgi.core-4.3.1.jar:/usr/share/jicofo/lib/sdes4j-1.1.3.jar:/usr/share/jicofo/lib/sdp-api-1.0.jar:/usr/share/jicofo/lib/slf4j-api-1.7.7.jar:/usr/share/jicofo/lib/slf4j-jdk14-1.7.7.jar:/usr/share/jicofo/lib/smack-3.2.2-jitsi-1-20151215.185921-3.jar:/usr/share/jicofo/lib/smackx-3.2.2-jitsi-1-20151215.185922-3.jar:/usr/share/jicofo/lib/tinder-1.2.3.jar:/usr/share/jicofo/lib/weupnp-0.1.4.jar:/usr/share/jicofo/lib/xml-apis-1.0.b2.jar:/usr/share/jicofo/lib/xmlpull-1.1.3.4a.jar:/usr/share/jicofo/lib/xpp3-1.1.4c.jar:/usr/share/jicofo/lib/zrtp4j-light-3.2.0-jitsi-1-20150723.002345-1.jar org.jitsi.jicofo.Main --domain=gofast-comm-dev.ceo-vision.com --secret=a4030ma --user_domain=gofast-comm-dev.ceo-vision.com --user_name=admin --user_password=a4030ma --subdomain=focus --host=localhost --port=5348

Supervisor

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX operating systems. In production, this results in a process called /usr/bin/supervisord:

[centos@gofast-comm-ceov-prod ~]$ ps aux | grep supervisord
root     16731  0.0  0.1 220916 12144 ?        Ss   Sep01   3:15 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf

Logs

Below are the log locations on VM2:

/var/log/nginx/

nginx

/var/log/onlyoffice/

OnlyOffice

-

PostreSQL

/var/log/redis/redis.log

Redis

-

Node.js

/var/log/jitsi/jicofo.log

Jicofo

/var/log/jitsi/jvb.log

Jitsi-videobridge

/var/log/messages

Network

/var/log/secure

Secure

/var/log/messages

Sshd

/var/log/messages

Su

/var/log/messages

Systemd

/var/log/wtmp

Wtmp

/var/log/zabbix/zabbix_agentd.log

Zabbix_agentd

/var/log/maillog

Maillog

/var/log/messages

NetworkManager

/usr/lib/rpm/rpm.log

Rpm

/var/log/yum.log

Yum

/var/log/messages

Lvm

/var/log/messages

Dracut

/var/log/messages

Ma1sd

/var/log/messages

Systemd-journald

/var/log/messages

Rabbitmqctl

/var/log/messages

Rabbitmq-server

/usr/lib/node_modules/npm/node_modules/socks

Yarn-error

/var/log/cron

Cron

/var/log/prosody/prosody.log

Prosody

/var/log/supervisor/supervisord.log

Supervisord

/var/log/messages

Setsebool

Supervision

Server Monitoring

On some offers, we automatically install a component to monitor the main server information.

This component is Newrelic (https://newrelic.com).

Note

In 2018, it is possible that we will change supervision providers

The main supervised information is as follows:

  • CPU Load

  • Disk IO

  • Ram Use

  • Available disk space

  • Network Usage

In production, this results in two /usr/sbin/nrsysmond` processes that make queries to the internet every 3 minutes:

[root@gofast3-integration ~]# ps aux | grep newrelic
root      1296  0.0  0.0 190024  1916 ?        Ssl  Aug16   0:01 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --port /tmp/.newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_docker=true
root      1303  0.0  0.0 504360  2788 ?        Sl   Aug16   0:34 /usr/bin/newrelic-daemon --agent --pidfile /var/run/newrelic-daemon.pid --logfile /var/log/newrelic/newrelic-daemon.log --port /tmp/.newrelic.sock --tls --define utilization.detect_aws=true --define utilization.detect_docker=true -no-pidfile
root     18457  0.0  0.0 112648   976 pts/1    R+   16:14   0:00 grep --color=auto newrelic
tomcat   23103 15.0 30.2 7071740 2410948 ?     Ssl  Aug22 228:27 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=1G -XX:-DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -XX:ParallelGCThreads=4 -XX:+UseParNewGC -Djavax.net.ssl.keyStore=/etc/pki/keystore -Djavax.net.ssl.keyStorePassword=a4030ma -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/alfresco -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/alfresco/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
root     29294  0.4  6.1 4298500 493744 ?      Sl   Aug16  41:08 java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.install.dir=/opt/solr -Xss256k -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -jar start.jar -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs --module=http
tomcat   31681  1.1 15.2 6935996 1217192 ?     Ssl  Aug16 114:46 /usr/lib/jvm/jre/bin/java -javaagent:/usr/share/tomcat/newrelic/newrelic.jar -server -Xss1024K -Xms1G -Xmx3G -XX:MaxPermSize=512M -XX:NewSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -Dbonita.home=/var/bonita -Dsysprop.bonita.db.vendor=mysql -Dbtm.root=/var/lib/tomcats/bonita/ -Dbitronix.tm.configuration=/var/lib/tomcats/bonita/conf/bitronix-config.properties -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/tomcats/bonita -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/var/lib/tomcats/bonita/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Data security (backup, DR,…)

The GoFAST platform brings together the strategic content of the organization. Data security must be based on an ‘architecture’ layer (RAID+SAN dual or clustering) coupled with a backup strategy.

Caution

Backup is therefore essential, as are restoration tests

The issue of allowable loss must be addressed, as must the time for restoration. This makes it possible to determine a backup strategy.

A) Remote backup of the entire platform:

  • By VM snapshot

B) Remote data backup only by vacation:

  • By saving application data

  • By full replication of data on a remote server (Disaster Recovery)

  • By saving files only

C) Remote data backup only in near real-time:

ON STUDY: CONTACT US

VM snapshot backup

In this case, the entire virtual machine is backed up.

Caution

It is recommended to take a daily snapshot of the VM that is out of business hours as there is an impact on performance (input/output or I/O). In addition, in order to ensure the integrity of the snapshot, the application may need to ‘freeze’ the VM for a certain period of time, depending on the technologies used.

Note

When CEO-Vision provides hosting (SaaS) to one of its partners, this type of backup is automatically included

Backup of application data

Once a day at the end of the day, all the information needed for the GoFAST platform is saved in /var/backup`.

Caution

It is strongly recommended that the maintainer mount /var/backup on a remote storage

To do this, using the Linux cron mechanism, the command /usr/bin/rsnapshotdaily is executed once a day. This mechanism calls a backup script created by CEO-Vision (/opt/ceo-vision/backup.sh`) which stores the required data in the ``/var/backup folder

If a retention period is set, it is possible to retrieve data from 1 or more days before in this folder /var/backup

The data saved are as follows:

  • the MySQL drupal database

  • the MySQL database alfresco

  • the ldap directory

  • the files of the Documentary Warehouse

  • Drupal sources

APPENDIX I: GoFAST Tree (VM1)

/opt/ceo-vision/

Application & Scripts CEO-Vision/GoFAST

/var/lib/tomcats/bonita

Applications

/var/lib/tomcats/alfresco

/opt/solr

/opt/libreoffice5.3

/var/backup

Backup space (openldap,alfresco, bonita, Drupal)

/var/lib/mysql

Application data

/var/lib/ldap

/var/www/d7

/var/alfresco

/etc/openldap

Configuration files

/etc/httpd

/etc/extra/browscap.ini

/etc/php-fpm.conf

/etc/my.cnf

/etc/crontab

/etc/newrelic

/var/lib/tomcats/alfresco/conf/tomcat_alfresco.conf

/etc/pki/tls/private/localhost.key /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/server-chain.crt

Certificates

APPENDIX II: Disaster Recovery “Minimal”

Note

This is an extension (option) of the GoFAST subscription, covering the upgrade of an additional environment.

In this case of DR Minimal, the principle is to bring back the backups created by GoFAST scripts (see “Backing up Application Data”), in a remote environment called standby.

The standby machine is a GoFAST installation as such. When the production environment is updated, the DR environment is updated by CEO-Vision.

Note

In order to ensure the integrity of Alfresco on the DR, the date of the saved files must match the date of the database snapshot. This is guaranteed by the script delivered with the GoFAST platform

Case 1) Alfresco remote backup is done in /var/backup

  • Import of the database

  • Copy /var/backup/…alfresco to /var/alfresco

  • Loading the LDAP part

**Case 2) The remote backup of Alfresco is done directly in the ** directory /var/alfresco

  • Import of the database

  • Loading the LDAP part

APPENDIX III: Remote Duplication of Files

It may be desirable to save a simple copy of the repository files on another server.

Caution

In this case, only the latest version of the files is saved. Metadata or comments are not saved

Network Reader

The first method is to use a backup software on the server to store the backups. This backup software must be able to back up a “network drive” or directly a Webdav server. In order to limit the bandwidth used and the machine resources it is best to make incremental or differential backups.

The ‘network drive’ has the following address:

https://url_de_la_gofast/alfresco/webdav

e.g.: https://gofast.ceo-vision.com/alfresco/webdav

Of course the username must be the user ‘adm’ who is the only user with access to all the documents of the platform.

Deployment of GoFAST Certificates:

Wildcards:

MAIN :

drop the folder containing the certificates to /opt

  • copy the certificate to /etc/pki/tls/certs/localhost.crt

  • copy the key to /etc/pki/tls/private/localhost.key

  • copy the server-chain to /etc/pki/tls/certs/server-chain.crt

COMM :

  • copy the certificate to /etc/pki/tls/certs/localhost.crt

  • copy the key to /etc/pki/tls/private/localhost.key

  • cat /etc/pki/tls/certs/gofast.crt /etc/pki/tls/private/gofast.key > /etc/pki/tls/certs/gofast.pem
    

Warning

Once done, restart the “nginx” service on the comm and the “httpd” service on the hand.

MAIN :
systemctl restart httpd
COMM :
systemctl restart nginx

Test the services (co-editing, co-editing lock, meeting, chat), fix if they no longer work.

Corrections to be made (if services malfunction):

Problem non-automatic connection to Element chat (COMM):

Commands to run on the COMM platform (in ssh) to fix the non-automatic chat connection problem.

cd /opt/gofast-comm
echo "Enter GoFAST MAIN URL (ex: gofast.ceo-vision.com):"
read url_main
java InstallCert $url_main
cp jssecacerts /etc/pki/ca-trust/extracted/java/cacerts
systemctl restart ma1sd

Problem Locks OnlyOffice and Non-Save Document (MAIN):

Commands to run on the MAIN platform (in ssh) to fix the lock problem that remains closed when a OnlyOffice co-edition is released (beware this problem causes that any edits on the document will not be saved)

cd /opt
echo "Enter GoFAST COMM URL (ex: gofast-comm.ceo-vision.com):"
read url_comm
java InstallCert $url_comm
cp jssecacerts /etc/pki/ca-trust/extracted/java/cacerts
echo "Warning la GED va redémarrer"
sleep 2
systemctl restart tomcat@alfresco

Meeting Certificates (COMM) issue:

Commands to run on the COMM platform (in ssh) to fix the problem of video conferencing at least three impossible. (with video stream):

Warning

The DMS will restart as a result of these commands, so make sure no one is working on it.

server_name=`grep -oP "(?<=domain: \').+?(?=\')" /etc/ma1sd/ma1sd.yaml`

echo "secret_key = mot de passe dans /etc/jitsi/jicofo/config "

read secret_key

rm -Rf /var/lib/prosody/*
cd /opt/gofast-comm/update

systemctl stop crond
prosodyctl stop
systemctl stop jibri
systemctl stop jitsi-videobridge
systemctl stop jicofo

#Generate certificates

prosodyctl cert generate auth.${server_name}

Warning

Enter the following information for the “prosodyctl cert generate auth.${server_name}” command:

4096

FR COMM

GOFAST

XMPP

auth.${server_name}

xmpp@auth.${server_name}

mv /var/lib/prosody/auth.$server_name.crt  /var/lib/prosody/auth.crt
mv /var/lib/prosody/auth.$server_name.key  /var/lib/prosody/auth.key
mv /var/lib/prosody/auth.$server_name.cnf  /var/lib/prosody/auth.cnf

ln -sf /var/lib/prosody/auth.crt /etc/pki/ca-trust/source/anchors/auth.crt

update-ca-trust extract -f

prosodyctl start

prosodyctl register videobridge auth.$server_name $secret_key
prosodyctl register jibri auth.$server_name $secret_key
prosodyctl register recorder recorder.$server_name $secret_key
prosodyctl register focus auth.$server_name $secret_key
prosodyctl mod_roster_command subscribe focus.$server_name focus@auth.$server_name

systemctl start jibri
systemctl start jitsi-videobridge
sleep 3
systemctl start jicofo
systemctl start crond

CEO-Vision Backup Policy

Machines hosted by Outscale

Customers’ machines hosted at Outscale are saved in the CEO-Vision repository at OVH.

Repo

OVH

Retention policy

7 catering points

Planning

Every day at 8.30 p.m.

Machines hosted by Scaleway

Customers’ machines hosted at Scaleway are saved in the CEO-Vision repository at OVH.

Repo

OVH

Retention policy

7 catering points

Planning

Every day at 04:00 a.m.

Machines hosted by OVH

Customers’ machines hosted at OVH are saved in the CEO-Vision repository at Scaleway.

Repo

SCAL

Retention policy

7 catering points

Planning

Every day at 8:30 p.m.