postgres borrows concepts from mysql (now mariadb)

postgres and mariadb are borth great databases, thanks all involved.

(how to setup postgres 12 on centos 8 (very very latest))

let’s dive into hit: how to postgres:

hostnamectl; # tested on
    Virtualization: kvm
  Operating System: CentOS Linux 8 (Core)
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.18.0-147.3.1.el8_1.x86_64
      Architecture: x86-64

psql --version; # what version is installed?
psql (PostgreSQL) 12.1

# view main config file
vim /var/lib/pgsql/12/data/postgresql.conf

# status of all installed postgres versions
# (yes it's crazy it is possible to run
# multiple versions of postgres at the same time on the same server)
systemctl status *postgres*
systemctl status postgresql-12.service

# example output
● postgresql-12.service - PostgreSQL 12 database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql-12.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-05-14 11:22:29 CEST; 1 weeks 4 days ago
     Docs: https://www.postgresql.org/docs/12/static/
  Process: 1199 ExecStartPre=/usr/pgsql-12/bin/postgresql-12-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 1226 (postmaster)
    Tasks: 9 (limit: 26213)
   Memory: 423.1M
   CGroup: /system.slice/postgresql-12.service
           ├─ 1226 /usr/pgsql-12/bin/postmaster -D /var/lib/pgsql/12/data/
           ├─ 1248 postgres: logger   
           ├─ 1252 postgres: checkpointer   
           ├─ 1253 postgres: background writer   
           ├─ 1254 postgres: walwriter   
           ├─ 1255 postgres: autovacuum launcher   
           ├─ 1256 postgres: stats collector   
           ├─ 1257 postgres: logical replication launcher   
           └─19273 postgres: dbuser dbname 127.0.0.1(59972) idle

# another possibility is
pg_isready
/var/run/postgresql:5432 - accepting connections

# restart
systemctl restart postgresql-12

# become user postgres
su - postgres

# change/reset password of user postgres
psql -c "alter user postgres with password 'StrongPassword'"
su - postgres -c "psql -c \"alter user postgres with password 'postgres'\""

# login as user postgres
# "postgres" user in postgres
# is like "root" in linux
psql -U postgres

# login and select dbname
psql -d dbname -U dbuser

# login to a specific host with ip 127.0.0.1
psql -U dbuser -h 127.0.0.1 dbname

# if user gets error: psql: error: could not connect to server: FATAL:  Ident authentication failed for user 
# edit this file
vim /var/lib/pgsql/12/data/pg_hba.conf

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     md5
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

# hit
ESC
# then
:wq # write and quit in vim
# restart postgres and try login again
systemctl restart postgresql-12

# try login as postgres root user
psql -U postgres
Password for user postgres: 
psql (12.1)
Type "help" for help.

postgres=# <- YESSSS! LOGGED IN :)

# list all databases
\l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 dbname    | dbuser   | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)


# create new user
CREATE USER dbuser WITH PASSWORD 'dbname';
# create new database
CREATE DATABASE dbname ENCODING 'UTF-8' TEMPLATE template0 OWNER dbuser;

# display short help
\h
# display extensive help
\?

General
  \copyright             show PostgreSQL usage and distribution terms
  \crosstabview [COLUMNS] execute query and display results in crosstab
  \errverbose            show most recent error message at maximum verbosity
  \g [FILE] or ;         execute query (and send results to file or |pipe)
  \gdesc                 describe result of query, without executing it
  \gexec                 execute query, then execute each value in its result
  \gset [PREFIX]         execute query and store results in psql variables
  \gx [FILE]             as \g, but forces expanded output mode
  \q                     quit psql
  \watch [SEC]           execute query every SEC seconds

Help
  \? [commands]          show help on backslash commands
  \? options             show help on psql command-line options
  \? variables           show help on special variables
  \h [NAME]              help on syntax of SQL commands, * for all commands

Query Buffer
  \e [FILE] [LINE]       edit the query buffer (or file) with external editor
  \ef [FUNCNAME [LINE]]  edit function definition with external editor
  \ev [VIEWNAME [LINE]]  edit view definition with external editor
  \p                     show the contents of the query buffer
  \r                     reset (clear) the query buffer
  \s [FILE]              display history or save it to file
  \w FILE                write query buffer to file

Input/Output
  \copy ...              perform SQL COPY with data stream to the client host
  \echo [STRING]         write string to standard output
  \i FILE                execute commands from file
  \ir FILE               as \i, but relative to location of current script
  \o [FILE]              send all query results to file or |pipe
  \qecho [STRING]        write string to query output stream (see \o)

Conditional
  \if EXPR               begin conditional block
  \elif EXPR             alternative within current conditional block
  \else                  final alternative within current conditional block
  \endif                 end conditional block

Informational
  (options: S = show system objects, + = additional detail)
  \d[S+]                 list tables, views, and sequences
  \d[S+]  NAME           describe table, view, sequence, or index
  \da[S]  [PATTERN]      list aggregates
  \dA[+]  [PATTERN]      list access methods
  \db[+]  [PATTERN]      list tablespaces
  \dc[S+] [PATTERN]      list conversions
  \dC[+]  [PATTERN]      list casts
  \dd[S]  [PATTERN]      show object descriptions not displayed elsewhere
  \dD[S+] [PATTERN]      list domains
  \ddp    [PATTERN]      list default privileges
  \dE[S+] [PATTERN]      list foreign tables
  \det[+] [PATTERN]      list foreign tables
  \des[+] [PATTERN]      list foreign servers
  \deu[+] [PATTERN]      list user mappings
  \dew[+] [PATTERN]      list foreign-data wrappers
  \df[anptw][S+] [PATRN] list [only agg/normal/procedures/trigger/window] functions
  \dF[+]  [PATTERN]      list text search configurations
  \dFd[+] [PATTERN]      list text search dictionaries
  \dFp[+] [PATTERN]      list text search parsers
  \dFt[+] [PATTERN]      list text search templates
  \dg[S+] [PATTERN]      list roles
  \di[S+] [PATTERN]      list indexes
  \dl                    list large objects, same as \lo_list
  \dL[S+] [PATTERN]      list procedural languages
  \dm[S+] [PATTERN]      list materialized views
  \dn[S+] [PATTERN]      list schemas
  \do[S]  [PATTERN]      list operators
  \dO[S+] [PATTERN]      list collations
  \dp     [PATTERN]      list table, view, and sequence access privileges
  \dP[itn+] [PATTERN]    list [only index/table] partitioned relations [n=nested]
  \drds [PATRN1 [PATRN2]] list per-database role settings
  \dRp[+] [PATTERN]      list replication publications
  \dRs[+] [PATTERN]      list replication subscriptions
  \ds[S+] [PATTERN]      list sequences
  \dt[S+] [PATTERN]      list tables
  \dT[S+] [PATTERN]      list data types
  \du[S+] [PATTERN]      list roles
  \dv[S+] [PATTERN]      list views
  \dx[+]  [PATTERN]      list extensions
  \dy     [PATTERN]      list event triggers
  \l[+]   [PATTERN]      list databases
  \sf[+]  FUNCNAME       show a function's definition
  \sv[+]  VIEWNAME       show a view's definition
  \z      [PATTERN]      same as \dp

Formatting
  \a                     toggle between unaligned and aligned output mode
  \C [STRING]            set table title, or unset if none
  \f [STRING]            show or set field separator for unaligned query output
  \H                     toggle HTML output mode (currently off)
  \pset [NAME [VALUE]]   set table output option
                         (border|columns|csv_fieldsep|expanded|fieldsep|
                         fieldsep_zero|footer|format|linestyle|null|
                         numericlocale|pager|pager_min_lines|recordsep|
                         recordsep_zero|tableattr|title|tuples_only|
                         unicode_border_linestyle|unicode_column_linestyle|
                         unicode_header_linestyle)
  \t [on|off]            show only rows (currently off)
  \T [STRING]            set HTML <table> tag attributes, or unset if none
  \x [on|off|auto]       toggle expanded output (currently off)

Connection
  \c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo}
                         connect to new database (currently "cyclos4")
  \conninfo              display information about current connection
  \encoding [ENCODING]   show or set client encoding
  \password [USERNAME]   securely change the password for a user

Operating System
  \cd [DIR]              change the current working directory
  \setenv NAME [VALUE]   set or unset environment variable
  \timing [on|off]       toggle timing of commands (currently off)
  \! [COMMAND]           execute command in shell or start interactive shell

Variables
  \prompt [TEXT] NAME    prompt user to set internal variable
  \set [NAME [VALUE]]    set internal variable, or list all if no parameters
  \unset NAME            unset (delete) internal variable

Large Objects
  \lo_export LOBOID FILE
  \lo_import FILE [COMMENT]
  \lo_list
  \lo_unlink LOBOID      large object operations
# use Ctrl+D
# or type
\q
# to quit/logoff postgres

# backup all databases to one file
pg_dumpall > $(date '+%Y-%m-%d-%H').backup.postgre.sql
# also compress it in one go
time pg_dumpall | gzip > $(date '+%Y-%m-%d-%H').backup_all_postgre_databases.sql.gz

# or if root is not allowed
# try with postgres default user "postgres"
pg_dumpall -U postgres > $(date '+%Y-%m-%d-%H').backup.postgresql

# restore all databases (src)
psql -f 2020-08-10-09.backup.postgresql postgres

# or copy the backup into a folder where gnu-linux user postgres is owner
su - root
mkdir /backup
cp -rv 2020-08-10-09.backup.postgresql /backup
chown -R postgres: /backup
su - postgres -c "psql -f /backup/2020-08-10-09.backup.postgresql;"

# backup/export/dump database "dbname"
# will create filename like: 
pg_dump -U dbuser -h localhost dbname > /backup/postgres/$(date '+%Y-%m-%d_%H_%M_%S')_dbname.sql

# how to import backup/database dump
psql -U dbuser dbname < /backup/postgres/2020-05-25_23_18_11_dbname.sql

# or if the dump is in dump.sql.gz packed format
gunzip -c /path/to/dump.sql.gz | psql -U postgres

# more documentation on export/import database backups
https://www.postgresql.org/docs/9.1/backup-dump.html

multi threaded backup and restore

$DB_TO_BACKUP=DatabaseName
# backup single database multi threaded (10x threads)
time su postgres -c "pg_dump -j 10 -Fd -Z 9 --file=$(date '+%Y-%m-%d').$DB_TO_BACKUP.postgres_dump_directory_format.compressed $DB_TO_BACKUP";

# restore single database multi threaded (10x threads)
$DB_TO_BACKUP=DatabaseName
createdb -U postgres $DB_TO_BACKUP
time su postgres -c "pg_restore -Fd -j 10 -d $DB_TO_BACKUP -U postgres $(date '+%Y-%m-%d').$DB_TO_BACKUP.postgres_dump_directory_format.compressed"

enable detailed logs

#!WARNING! THIS WAS TESTED FOR OLDER RHEL7 AND CENTOS!

# manual editing
vim /var/lib/pgsql/12/data/postgresql.conf; # postgres main config file

# semi-automatic editing via sed
# optional: postgres: enable detailed logging
POSTGRES_MAIN_CONFIG="/var/lib/pgsql/12/data/postgresql.conf"

sed -i 's/#log_statement = [^"]*/log_statement = 'all'/g' $POSTGRES_MAIN_CONFIG

sed -i 's/#logging_collector = [^"]*/logging_collector = on/g' $POSTGRES_MAIN_CONFIG

sed -i 's/#log_connections = [^"]*/log_connections = on/g' $POSTGRES_MAIN_CONFIG

sed -i 's/#log_disconnections = [^"]*/log_disconnections = on/g' $POSTGRES_MAIN_CONFIG

sed -i 's/#log_error_verbosity = [^"]*/log_error_verbosity = default/g' $POSTGRES_MAIN_CONFIG

# even more verbose logging of details
# this is VERY VERY detailed
sed -i 's/#log_error_verbosity = [^"]*/log_error_verbosity = verbose/g' $POSTGRES_MAIN_CONFIG

systemctl restart postgresql-12.service; # restart to make config changes active

# logs should now output here
==> /var/lib/pgsql/12/data/log/postgresql-Mon.log <==
==> /var/lib/pgsql/12/data/log/postgresql-Tue.log <==

# let's monitor those
vim /scripts/monitor.sh

#!/bin/bash
# version: 1.1
# description: this script will allow to see changes on the system in real time, it allows monitoring multiple log files at the same time
# usage: /path/to/script/monitor.sh
# monitoring all files below /var/log
find /var/log/ -type f \( -name "*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} + &
# monitoring postgres logs (please enable logging in /var/lib/pgsql/12/data/postgresql.conf manually
find /var/lib/pgsql/12/data/log/ -type f \( -name "*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} + &
# monitoring all files.log below /var/opt
find /var/opt/ -type f \( -name "*log*" \) ! -path *.gz* -exec tail -n0 -f "$file" {} +

script: display information about the installed postgres instance: show info about postgres database, files, config and usage stats

  • … postgres: installed version?
    su - root
    echo "... postgres: installed version?"
    su - postgres -c "psql -d postgres -c \"SELECT version();\""
    
    echo "... postgres: where is the main config file?"
    su - postgres -c "psql -d postgres -c \"SHOW config_file;\""
    
    echo "... postgres: where is the main data directory? (where files are stored)"
    su - postgres -c "psql -d postgres -c \"SHOW data_directory;\""
    
    echo "... postgres: usage stats?"
    su - postgres -c "
    psql -d postgres -c \"
    SELECT
        'Database Name' AS database,
        pg_database.datname AS database,
        pg_size_pretty(pg_database_size(pg_database.datname)) AS size,
        pg_stat_database.numbackends AS connections,
        pg_stat_database.xact_commit AS commits,
        pg_stat_database.xact_rollback AS rollbacks,
        pg_stat_database.blks_read AS blocks_read,
        pg_stat_database.blks_hit AS blocks_hit,
        pg_stat_database.tup_returned AS rows_returned,
        pg_stat_database.tup_fetched AS rows_fetched,
        pg_stat_database.tup_inserted AS rows_inserted,
        pg_stat_database.tup_updated AS rows_updated,
        pg_stat_database.tup_deleted AS rows_deleted
    FROM
        pg_database
    JOIN
        pg_stat_database
    ON
        pg_database.datname = pg_stat_database.datname
    WHERE
        pg_database.datistemplate = false
    ORDER BY
        pg_database.datname;\"
    "
    

alternativels download the script: postgres_info.sh.txt

move postgres db to differen harddisk or partition or filesystem

systemctl stop postgresql

rsync -r -vv --update --progress /var/lib/postgresql/12/main/ /media/user/data/postgres

sed -i.bak "s|data_directory = '/var/lib/postgresql/12/main'|data_directory = '/media/user/data/postgres'|" /etc/postgresql/12/main/postgresql.conf

# give user postgres access
chown -R postgres:postgres /media/user/
chmod 777 /media/user/
chmod 777 /media/user/data/

# or postgres will complain
chmod 0750 /media/user/data/postgres

# make sure to enable detailed logging (see above)
# and watch all logs closely during startup process as errors might occur
systemctl start postgresql

benchmarks?

  • not sure how “legit” it is, this benchmark will create create a db test_benchmark with 10x tables and create 10x dump.sql and import that one by one…
    • postgres_bench_v2.sh.txt
      • result: inside a kvm vm this bench took 2m56.246s which means
      • echo “scale=5; 10000*10/(176)”|bc
        568 inserts per sec (IOPS?) (with no significant usage of CPU, RAM or DISC this means: the bottleneck is somewhere else)

liked this article?

  • only together we can create a truly free world
  • plz support dwaves to keep it up & running!
  • (yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
  • really really hate advertisement
  • contribute: whenever a solution was found, blog about it for others to find!
  • talk about, recommend & link to this blog and articles
  • thanks to all who contribute!
admin