Ubuntu + Debian: how to pip (python package manager) cheat sheed how to examples and getting started with robot + selenium gui testing framework – nokia’s eclipse red vs vscode extension (how to eclipse+pydev + virtual python environment) (error: “externally-managed-environment” “This environment is externally managed”)

29.Sep.2024

devPython, software testing and quality assurance

imho it’s recommended to solve things without 3rd party packages (which might possible be neglected (no more security updates) or even hijacked after some time)

it is sometimes agreed upon to extend python’s capabilities with 3rd party packages like psutil (for system and process monitoring)

installation of pip: multiple possibilities X-D

how to install pip (python package manager) (github src 99% python):
then continue wtih gecko driver install (download the binary and make it accessible)

# tested with Ubuntu
su - root
apt update
# should work
apt install pip
# but more correct, package is called
apt -y install python3-pip

apt show python3-pip
Package: python3-pip
Version: 23.0.1+dfsg-1
Priority: optional
Section: python
Source: python-pip
Maintainer: Debian Python Team <(does not want spam)>
Installed-Size: 6,838 kB
Provides: pip
Depends: ca-certificates, python3-distutils, python3-setuptools, python3-wheel, python3:any
Recommends: build-essential, python3-dev (>= 3.2)
Breaks: python-pip
Replaces: python-pip
Homepage: https://pip.pypa.io/en/stable/
Download-Size: 1,325 kB
Description: Python package installer

pip is the Python package installer.

It integrates with virtualenv
+ doesn't do partial installs
+ can save package state for replaying
+ can install from non-egg sources,
+ and can install from version control repositories.

This is the Python 3 version of the package.

search for a package here https://pypi.org/

AS NON ROOT USER (NO ROOT REQUIRED) install a package:

pip install packagename

list all installed packages:

pip list|less

update all packages:

# strange enough there is no pip3 update && pip3 upgrade, but a rather complex construct must be used?
pip3 list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip3 install -U

getting started with robot + selenium gui testing framework

as non-root user: install those 3 packages

pip3 install robotframework
pip3 install robotframework-seleniumlibrary
pip3 install webdrivermanager

Debian

Debian is an EXCELLENT OS, it is the basis on which Ubuntu builds, hopefully canonical will continue to backport and support Debian, because without a strong fundament, it’s not possible to build a good smooth runnig secure OS.

Debian is way more minimalistic than Ubuntu and runs on machines with as little as 512MB RAM.

for some reason the robot + selenium setup is not as easy on Debian as on Ubuntu and the use of virtual environment is required

(there is a python3-selenium package, but nopython3-robotframework package? why?)

hostnamectl; # tested on
Operating System: Debian GNU/Linux 12 (bookworm) 
Kernel: Linux 6.1.0-28-amd64
Architecture: x86-64

apt -y install python3.11-venv
apt -y install python3.11-full
apt -y install pipenv

# this will create a new folder called "virtualenvironment"
python3 -m venv virtualenvironment

# activate the environment, making the packages available to python3
source virtualenvironment/bin/activate

# install pip packages
pip install robotframework robotframework-seleniumlibrary webdrivermanager

how to make virtual python environment known to eclipse+pydev

windows -> preferences -> pydev -> python interpreter

within ./virtualenvironment/bin/python there are soft links to python interpreter under different LinkNames (just select one of them for example “python”)

“python” -> python3
“python3” -> /usr/bin/python3
“python3.11” -> python3

ultimately all link to the same file: file /usr/bin/python3.11
/usr/bin/python3.11: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=fbd431f5b0c39cedcac43d997b06c18af7d953a7, for GNU/Linux 3.2.0, stripped

gecko driver install

wget https://github.com/mozilla/geckodriver/releases/download/v0.35.0/geckodriver-v0.35.0-linux64.tar.gz

tar fxvz geckodriver-v0.35.0-linux64.tar.gz

chmod +x geckodriver

# as root
ln -sv /home/user/software/geckodriver /usr/local/bin/

# log off root, continue as non-root user
Ctrl+D

create test script to test if selenium works

vim selenium_test.py

with this content

# does work
from selenium import webdriver
import time
from time import sleep
driver = webdriver.Firefox()
driver.get("http://dwaves.org")
time.sleep(8)
driver.quit()
print("test success")

run it:

python3 ./selenium_test.py

if this error pops up: users Firefox profile cannot be loaded. It may be missing or inaccessible.

it is because firefox was installed via snap and not apt

uninstall it via snap and re-install it with apt

su - root
snap remove firefox;

install -d -m 0755 /etc/apt/keyrings;

wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- | sudo tee /etc/apt/keyrings/packages.mozilla.org.asc > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main" | sudo tee -a /etc/apt/sources.list.d/mozilla.list > /dev/null;

echo '
Package: *
Pin: origin packages.mozilla.org
Pin-Priority: 1000

Package: firefox*
Pin: release o=Ubuntu
Pin-Priority: -1' | sudo tee /etc/apt/preferences.d/mozilla;

apt update && sudo apt remove firefox;

apt install firefox;
# try re running the test
python3 ./selenium_test.py

if the dev user sees the website opening and browser closing after 8 sec

celebrate! 😀

what should also work…

open website
enter “selenium” into search textinput
hit enter

create script

vim ./selenium_test_submit_form.robot

# test_suite.robot v1.0
# open duckduckgo in firefox browser (installed via apt not snap)
# enter "selenium" in search field
# submit the form
*** Settings ***
Library    SeleniumLibrary

*** Variables ***
${SEARCH_TERM}    selenium

*** Test Cases ***
Open Dwaves And Search selenium
    Open Browser    https://dwaves.org    Firefox
    Input Text      name=s                ${SEARCH_TERM}
    Submit Form     searchform
    Sleep           5 seconds
    Close Browser

*** Test Cases ***
Open DuckDuckGo And Search Selenium
    Open Browser    https://duckduckgo.com  Firefox
    Input Text      name=q                  ${SEARCH_TERM}
    Submit Form     searchbox_homepage
    Sleep           5 seconds
    Close Browser

WARNING! THE MASSIVE AMOUNTS OF SPACES (AT LEAST 4?) IS ON PURPOSE AND IT MIGHT NOT RUN WITHOUT AT LEAST 4 SPACES!

it might be useful to get more verbose output in a specified dir:


mkdir output
robot -d output --loglevel DEBUG selenium_test_submit_form.robot

externally-managed-environment

The error `externally-managed-environment` typically occurs when user trying to install packages in a Python environment that is managed by an external package manager like `apt` on Ubuntu or `brew` on macOS.

Here are some steps to resolve this issue:

### 1. Use a Virtual Environment

The best practice is to use a virtual environment for user Python projects:

– **Create a virtual environment:**
“`bash
python3 -m venv myenv
“`

– **Activate the virtual environment:**
– On Windows:
“`bash
myenv\Scripts\activate
“`
– On macOS/Linux:
“`bash
source myenv/bin/activate
“`

– Now, install `robotframework`:
“`bash
pip install robotframework
“`

### 2. Use `–user` Flag

If user can’t or don’t want to use a virtual environment, user can try installing the package for the current user:

“`bash
pip install –user robotframework
“`

### 3. Check Python Installation

– Ensure user using the correct Python version. Sometimes, system Python might be linked to Python 2, while user need Python 3 for `robotframework`.

– Check users Python version:
“`bash
python –version
“`
or
“`bash
python3 –version
“`

### 4. Use System Package Manager

If user on a system where Python was installed via a package manager:

– **Ubuntu/Debian:**
“`bash
sudo apt-get install python3-robotframework
“`

– **Fedora:**
“`bash
sudo dnf install python3-robotframework
“`

– **macOS with Homebrew:**
“`bash
brew install robotframework
“`

### 5. Modify `pip` Configuration

If user in an environment where user can’t change the Python setup:

– You might need to modify `pip` configuration to allow installations in an externally managed environment. However, this is not recommended as it can lead to system instability:

“`bash
pip config set global.break-system-packages true
“`

Then try installing again:
“`bash
pip install robotframework
“`

### Note:

– Modifying system Python environments can lead to issues with system stability or updates. Always prefer using virtual environments for project-specific dependencies.

– If user working in a shared or managed environment, consult with users system administrator or follow the guidelines provided for managing Python packages.

virtual environments? why?

g “python: please explain virtual environment, and instead of ‘you’ use ‘user'”
=== grok.sh v1.0 ===
A **virtual environment** in Python is a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages, different from the system-wide Python installation. Here’s a detailed explanation:

### Purpose of Virtual Environments

– **Isolation**: Virtual environments allow users to isolate project dependencies. This means each project can have its own dependencies, regardless of what dependencies other projects have.

– **Dependency Management**: They help manage and segregate different versions of packages for different projects. This prevents conflicts where one project might require a different version of a library than another.

– **Reproducibility**: By specifying the exact versions of libraries needed, virtual environments ensure that the project can be set up in the same way on different machines, enhancing reproducibility.

– **System Cleanliness**: Users can keep their global Python environment clean, avoiding clutter from project-specific packages.

### How Virtual Environments Work

1. **Creation**: Users create a virtual environment using tools like `venv` (built into Python 3.3+), `virtualenv`, or `conda` for Anaconda users. For example:
“`bash
python -m venv my_project_env
“`
This command creates a directory named `my_project_env` containing a copy of the Python interpreter, the standard library, and various supporting files.

2. **Activation**: To use the virtual environment, users must activate it:
– On Windows:
“`bash
my_project_env\Scripts\activate
“`
– On macOS and Linux:
“`bash
source my_project_env/bin/activate
“`
Once activated, any Python commands will use the environment’s Python interpreter and libraries.

3. **Installation of Packages**: With the environment active, users can install packages using `pip`:
“`bash
pip install package_name
“`
These packages will only be available within this virtual environment.

4. **Deactivation**: To return to the global Python environment:
“`bash
deactivate
“`

### Benefits

– **Project Specificity**: Each project can have its own set of libraries, which might not be compatible with other projects.
– **Version Control**: Users can control which version of Python and which versions of libraries are used for each project.
– **Security**: By not installing packages globally, users reduce the risk of system-wide security vulnerabilities.

### Considerations

– **Management**: Users need to manage multiple environments, which can become complex with many projects.
– **Space**: Each environment takes up disk space, although this is usually not a significant concern with modern storage capacities.

Virtual environments are an essential tool for Python developers to maintain clean, reproducible, and isolated project environments, ensuring that development and deployment processes are smooth and conflict-free.

why pipx?

**Purpose**: `pipx` is designed for installing and running end-user applications written in Python. It’s essentially a wrapper around `pip` that provides an isolated environment for each application.
– **Installation Scope**:
– `pipx` installs applications into their own isolated virtual environments, which means:
– Each application has its own dependencies, preventing conflicts between different applications.
– The system-wide Python environment remains clean, as `pipx` does not install packages globally.

eclipse + virtual python environments?

nokia’s eclipse red vs vscode extension

long story short:

vscode:
[+] step debugging script.robot works
[+] follow variable definition with ctrl key works
eclipse2020 or eclipse2024+red:
[-] not developed since 2020 (probably because it is in competition with vscode and M$ bought nokia in 2014)
[-] editor did not work yet (tried 2024 version, tried 2020 version with this url https://sourceforge.net/projects/red-robot-editor/files/)

Debian’s readme

error: “externally-managed-environment” “This environment is externally managed”

cat /usr/share/doc/python3.11/README.venv

Installing Python Packages in Debian
====================================

It is recommended to let Debian’s package managers manage Python packages in /usr/lib/ and /usr/share/

Python applications

If user needs to install a Python application (or version) that isn’t packaged in Debian, Debian recommends that user installs it with pipx (in the “pipx” Debian package)

pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.

Python library modules
———————-

If user needs to install a Python library module (or version) that isn’t packaged in Debian, Debian recommends installing it into a virtualenv, where possible.

User can create virtualenvs with the venv Python stdlib module (in the “python3-venv” Debian package) or the virtualenv Python 3rd-party tool (in the “virtualenv” Debian package).

Both of these will create an isolated environment, with a copy of pip in it.

After activating the environment, user can install python applications and library modules into the virtual environment.

# e.g. instead of running:
pip install --user foo
# run:
mkdir -p ~/.venvs
python3 -m venv ~/.venvs/foo
~/.venvs/foo/bin/python -m pip install foo

If needed, the isolated environment can also have access to system Python modules, with the “–system-site-packages” flag.

Installing things system-wide
—————————–

Because Debian declares its Python install to be EXTERNALLY-MANAGED [0], pip (and other installers) will refuse to install packages system-wide.

Installation is only possible in virtual environments or separate Python installs.

[0]: https://peps.python.org/pep-0668/

This is because Python package installers (like pip) are unaware of the constraints that APT-managed packages have on libraries and versions.

See PEP-668 for a full discussion of the problems that can occur when multiple installers operate on the same Python install.

This can be overriden by passing the –break-system-packages option to pip.

User does this at own risk: pip may break Python modules that part of your Debian system depends on.

This option can also be specified by exporting PIP_BREAK_SYSTEM_PACKAGES=1 or configuring the following in ~/.config/pip/pip.conf or /etc/pip.conf:

[global]
break-system-packages = true

User can also override this system-wide by removing
/usr/lib/python3.*/EXTERNALLY-MANAGED

Again, user does this at own risk, with the understanding that Debian-provided Python modules may be broken by pip and other installers.

best way to do this is to move it with dpkg-divert, which will survive security updates

# dpkg-divert –rename –add /usr/lib/$(py3versions -d)/EXTERNALLY-MANAGED

A clean option is to install user’s own Python (from source) in /usr/local, that isn’t EXTERNALLY-MANAGED.

Links + creditz

how to manually install webdriver https://www.youtube.com/watch?v=pNo27xOap_w
https://www.omgubuntu.co.uk/2022/04/how-to-install-firefox-deb-apt-ubuntu-22-04

liked this article?

only together we can create a truly free world
plz support dwaves to keep it up & running!
(yes the info on the internet is (mostly) free but beer is still not free (still have to work on that))
really really hate advertisement
contribute: whenever a solution was found, blog about it for others to find!
talk about, recommend & link to this blog and articles
thanks to all who contribute!

admin