Python & Server Scripts

Mysql - Empty Field with None value Error

Mysql error

Mysql - Empty Field with None value Error

Mysql is set to strict mode, remove this.

SSH Log in as root

Open the file /etc/my.cnf

add the line

sql_mode=

save the file

Restart mysql service

/scripts/restartsrv_mysql

PDF to image

yum install poppler-utils

Also include wkhtmltoimage

FFMPEG

Do not install using the YUM package as the FFMPEG is NOT up to date.

Instead we need to install a version with frei0r library.

https://sysadminxpert.com/install-ffmpeg-on-centos-7/

FFMPEG Packages.

https://johnvansickle.com/ffmpeg/

Step 1: Download FFmpeg build tar file

Switch to /opt directory and run below command to download FFmpeg build.

$ cd /opt $ sudo wget https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz

Step 2: Verify FFmpeg tar file using md5sum

$ sudo wget https://johnvansickle.com/ffmpeg/builds/ffmpeg-git-amd64-static.tar.xz.md5

$ md5sum -c ffmpeg-git-amd64-static.tar.xz.md5

Step 3: Untar the FFmpeg build

$ sudo tar xvf ffmpeg*.xz

$ cd ffmpeg-*-static

$ ll

Step 4: Create symlink for ffmpeg and ffprobe binaries

NOTE: I also did this for /usr/bin/

$ sudo ln -s "${PWD}/ffmpeg" /usr/local/bin/

$ sudo ln -s "${PWD}/ffprobe" /usr/local/bin/

Test

which ffmpeg

which ffprobe

Install Frei0r

sudo yum -y install frei0r-plugins

Frei0r commands

https://gstreamer.freedesktop.org/documentation/frei0r/frei0r-filter-softglow.html?gi-language=c#frei0rfiltersoftglow-page

Youtube-dl

I used this, however you can just use python pip, but will need python first.

sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl

sudo chmod a+rx /usr/local/bin/youtube-dl

Or python

sudo pip install --upgrade youtube_dl

Remove Background

Overview: We need to ensure we are up to date with systems (Centos, failed on old version).

We need to have gcc - linux C++ at a reasonable version.

Python3.8 and the correct Pytorch for it to run. See here:

https://github.com/danielgatis/rembg

python3.8 pip install rembg

Pytorch (I used an older version which is available on the website)

# CPU only

pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

https://pytorch.org/

https://pytorch.org/get-started/previous-versions/

Python3.8

Do not use YUM - as it installed an old version of Python.

Step 1 – Install Required Packages

Use the following command to install prerequisites for Python before installing it.

sudo yum install gcc openssl-devel bzip2-devel libffi-devel zlib-devel

Step 2 – Download Python 3.8

Download Python using following command from python official site. You can also download the latest version in place of the specified below.

cd /opt

wget https://www.python.org/ftp/python/3.8.12/Python-3.8.12.tgz

Now extract the downloaded package.

tar xzf Python-3.8.12.tgz

Step 3 – Compile Python Source

Use below set of commands to compile python source code on your system using altinstall.

make altinstall is used to prevent replacing the default python binary file /usr/bin/python.

I used make install

cd Python-3.8.12

sudo ./configure --enable-optimizations

sudo make altinstall

Now remove downloaded source archive file from your system

sudo rm Python-3.8.12.tgz

Step 4 – Check Python Version

Check the latest version installed of python using the following command.

python3.8 -V

Potrace - SVG tracer

http://potrace.sourceforge.net/

Download and install it.

Exif tools

Runs via per script see here.

https://www.geeksforgeeks.org/installing-and-using-exiftool-on-linux/

rabbitmq

https://www.vultr.com/docs/how-to-install-rabbitmq-on-centos-7/

Access interface here

http://173.231.229.92:15672/

Command for terminal on windows machine - use utilities from start menu - RabbitMQ start / stop

Ensure permissions are set chmod +x rabbit_consumers.sh

Also do it with rabbitmq_media_consumer.php

However you just need to CANCEL the running consumer script which was started by the initial bash to update the consumer.

On the server run $top to see running processes. There will be a CGI script that is running from the bash script called from cron - you must cancel that script, which is the rabbitmq_media_consumer.php find the PID with this command from the bash

ps -ef|grep /home/audasm5/public_html/php/rabbitmq_media_consumer.php |grep -v grep

ps -aux | grep /home/ozhousescom/public_html/php/rabbitmq_media_consumer.php

Do not use sudo when inside a web domain.

sudo kill pid

Then kill that id - so $kill -p <PID>

Ensure the cron job is set

/home/audasm5/public_html/php/rabbit_consumers.sh audasm5

Adding a new vhost.

Add via the rabbitmq admin panel and also add the admin user.

Also add the domain folder name to mysql vhosts in config,. rabbitmq_vhosts in the config settings.

The name is ozhousescom eg /home/ozhousescom - don't forget to add the CRON job to be running it.

So we need to add this folder name as a variable at the end of the call to the shell script.

*	*	*	*	*	`/home/ozfooty/public_html/php/rabbit_consumers.sh ozfooty`

Also ensnure that the sync_check is turned on in the config file if using. force_sync_check is for presentations to sync.

Errors can occur if the mail box is not configured correctly in the config table of the database.

Xvfb headless browser

ps -aux | grep Xvfb

sudo kill pid

How to Install Xfvb on CentOS 7 / 8 or RHEL 7 / 8

# yum install xorg-x11-server-Xvfb

To save a file in Vim / vi, press Esc key, type :w and hit Enter key

One can save a file and quit vim / Vi by pressing Esc key, type :x and hit Enter key

Create systemd xvfb service on CentOS 7 / 8 or RHEL 7 / 8

# vi /etc/systemd/system/xvfb.service

THIS CAN ONLY BE RUN AS ROOT. NO AUDIO YET?!

When sending a queued media request from rabbitmq the responding php script video_capture_server.php will write a file to run the render. It will then wait until a background process which has been started by cron runs that script and REMOVES the run_ffmpeg scripts. When they are removed it responds to rabbitMq as complete thus keeping the queue process in tact.

Paste code below

[Unit]

Description=Xvfb headless plotting

After=network.target

[Service]

User=root

ExecStart=/usr/bin/Xvfb :0 -screen 0 1280x1024x24

[Install]

WantedBy=multi-user.target

Alias=Xvfb.service

Alias=xvfbd.service

Make excecutabe /etc/systemd/system/xvfb.service and reload systemd daemon with command

# chmod +x /etc/systemd/system/xvfb.service

# systemctl daemon-reload

Enable and start xvfb.service

# systemctl enable xvfb.service

# systemctl start xvfb.service

To check status of xvfb

# systemctl status xvfb.service

https://serverdiary.com/linux/how-to-install-xvfb-and-create-systemd-xvfb-service-on-centos-7-8-or-rhel-7-8/

Install CHROME

vi /etc/yum.repos.d/google-chrome.repo

add the following

[google-chrome]

name=google-chrome

baseurl=http://dl.google.com/linux/chrome/rpm/stable/$basearch

enabled=1

gpgcheck=1

gpgkey=https://dl-ssl.google.com/linux/linux_signing_key.pub

yum install google-chrome-stable

https://www.tecmint.com/install-google-chrome-on-redhat-centos-fedora-linux/

How To General

# This is a sample build configuration for PHP.

# Check our guides at https://confluence.atlassian.com/x/e8YWN for more examples.

# Only use spaces to indent your .yml configuration.

# -----

# You can specify a custom docker image from Docker Hub as your build environment.

image: samueldebruyn/debian-git

pipelines:

default:

- step:

script:

- apt-get update

- apt-get -qq install git-ftp

- git config git-ftp.syncroot .

- git ftp init --force --user $FTP_USER_**** --passwd $FTP_PASSWORD_**** ftp://quickweb.com.au/public_html/

************BITBUCKET NOTES

Take the .git-ftp.log file of the copied directory and ensure that it is inside the target of the NEW upload. This will sync it. Then there is no need to init --force

************REFUSED to load css and js files

Ensure that the htaccess file does not have a superfluous / slash at the end of url site domain. Often caused by WRONG url.

OPened the htaccess file and remove the extra slashes from where the URL is. If this is still happening then we need to edit the output of the output of the htaccess files.

It occurred with the copying of the htaccess file when I loaded it from the pop-up window in the admin.

I added these to the .htaccess file - however this is not good enough

AddType text/css .css

AddType application/javascript .js

************CUSTOM PAGE /PRESENTATION DOES NOT SAVE

Ensure that the php.ini settings are updated with post max vars, max upload, execution time etc

************ADD CRON JOB

#CRON JOB

*/5 * * * *

php /home/propagatebus/public_html/php/crud_send_campaign.php cron_messages_campaigns 6930fabd5c462dccce120e512aa12693ccb025bb151e60cd247a885119cda6a5

************REMOVE FOLDERS AND DIRECTORIES - without removing the parent

rm -rf media/*

************REMOVE FOLDERS AND DIRECTORIES - INCLUDING PARENT

To remove a folder with all its contents (including all interior folders):

rm -rf /path/to/directory

*NOTCE no first slash

rm -rf path/to/directory

To remove all the contents of the folder (including all interior folders) but not the folder itself:

rm -rf /path/to/directory/*

rm -rf /media/*

rm -rf /images/made_images/*

************DUPLICATE SITE

FOR subdomain we need to create the folder first via cpanel - just write the name "howto" for example in subdomain field, nothing else - it takes care of it.

login as root and copy directory eg, if you want the CONTENTS and not include the public_html folder for subdomains etc - then use a star/wildcard at the end

cp -R /home/pixsellio/public_html/ /home/nextgenmortgages/

cp -r /home/pixsellio/public_html/ /home/harryrolfe/

cp -r /home/pixsellio/public_html/ /home/propagatebus/

cp -r /home/pixsellio/public_html/* /home/pixsellio/test.pixsell.io

cp -r /home/pixsellio/public_html/* /home/audasm5/howto.audasmedia.com.au

Then change the file owner/groups. First recursively change public_html/ then change public_html/ back leaving all content changed.

you can check ownership of files with command $ls -l

chown -R pixsellio:pixsellio test.pixsell.io/

chown -R pixsellio:nobody test.pixsell.io/

chown -R nextgenmortgages:nextgenmortgages public_html/

chown nextgenmortgages:nobody public_html/

chown -R audasm5:audasm5 howto.audasmedia.com.au/

chown audasm5:nobody howto.audasmedia.com.au/

chown -R propagatebus:propagatebus public_html/

chown propagatebus:nobody public_html/

update.htaccess

update config.php

We may need to move the "media" directory from an old/corrupted site use this command - dont forget to update the ownership. Also the "" quotes areound the cp are to allow recursive by doing the command as root.

"cp" -rf /home/propagatebus/quarantine/quarantine_1675464720/home/propagatebus/public_html/media /home/propagatebus/public_html/

copy user.ini and php.ini

************BITBUCKET

Failure in image - debryuen - switched to bitnami

Failed in remote .git-ftp.log files - change the hash SHA 1 in the remote server git-ftp.log to one that previous worked an then resubmit - the previous one that worked can be found in the actual edited file on bitbucket at the top.

************REMOVE BACKGROUND

https://github.com/danielgatis/rembg/

requires c++ build tools on windows to work

pytorch

https://pytorch.org/

python3.8 install on centos

https://computingforgeeks.com/how-to-install-python-3-on-centos/

************INSTALL SCRIPTS LINUX CENTOS

E.G.

yum install poppler-utils

also you can use pip3

command for pytorch

pip3.8 install --trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pytorch.org --trusted-host=download.pytorch.org --trusted-host=files.pypi.org --trusted-host=files.pytorch.org torch==1.10.0+cpu torchvision==0.11.1+cpu torchaudio==0.10.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html

PYTHON

in the new python folder - go to modules/setup and uncomment the lines for ssl about half way down

./configure --with-openssl=/usr/local/openssl --enable-optimizations

sudo make altinstall

OPENSSL

./config shared --prefix=/usr/local/openssl --openssldir=/usr/local/openssl

************FWM freds image magick imagemagick scripts

Change the files to have EOL format for unix. This means LF only and now CRLF

<?php

$pattern = __DIR__ ."/php/utils/imagemagick_scripts/new_need_to_be_added/*";

$files = glob($pattern);

// Can do multiple file types like this:

//$files = glob('folder/*.{php,txt}', GLOB_BRACE);

foreach($files as $file) {

if(is_dir($file))continue;

isa_normalize($file);

}

function isa_normalize ($filename) {

echo "Convert the ending-lines of $filename... ";

$string = @file_get_contents($filename);

if (!$string) {

echo "Could not convert the ending-lines : could not load the file.";

return false;

}

//Convert all line-endings from Windows to Unix

// $string = str_replace("\n", "\r\n", $string);

$string = str_replace("\r\n", "\n", $string);

$string = str_replace("\r", "\n", $string);

file_put_contents($filename, $string);

echo "Ending-lines converted.";

return true;

}