smemory_TechSomething

pfSense wireguard VPN with Mullvad

2026-03-13T00:00:00+00:00

Aim #

We have a pfSense box and we want some subnets going out only via VPN so the traffic won't exit our ISP ip address.

We'll configure a wireguard VPN using Mullvad.

We'll also add a killswitch to ensure no unwanted traffic from the subnets goes out via WAN interface.

Configure the VPN #

these guides are perfect:

please ignore the section "Adding a kill switch" from alexmoch's blog because for me it was not working.

Configure the killswitch #

LAN: our interface we want to route out of the vpn
se1_gw: the vpn gateway

in Firewall --> Rules --> YourSubnet you'll have a rule like this:

Action: pass
Interface: INT
Address Family: ipv4
Protocol: any

Source: LAN subnets

Destination: Any

Log: yes


ADVANCED:

Tag: ExitViaVpn

Gateway: se1_gw (your mullvad gateway name, I'll use mullvad's guide names)

Note the Tag and Gateway options:

Tag: tags our packets so we can filter them using that

Gateway: instructs the rule to use a different gateway than the default one.

Now for the killswitch, we'll head to Firewall --> Rules --> Floating
and add a new rule:

Action: block
Quick: yes
Interface: WAN
Direction: any
Address Family: IPV4
Protocol: any

Source: any
Destination: any

Log: yes

ADVANCED:

Tagged: ExitViaVpn

This rule allows us to block all the traffic from the "VPN subnet" that tries to use the WAN interface, thus exiting not via the VPN.

This is needed because when the VPN fails, and this also happened with openVPN, the "Gateway" option in the firewall rule gets ignored and the traffic goes out via the default gateway.

HP Z440 Workstation Upgrade and Improvements

2026-03-11T00:00:00+00:00

Why #

I had some 32GB DDR4 registered dimms from old hardware and I wanted to do something with those.

I've found that the best bet would be an X99 platform 0, but the Aliexpress boards would only have 4 dimm slots and other used mainboards would need a complete system around: PUS, case, etc..

The HP Z440 1 is a workstation from 2014 that:

supports 8 dimm slots up to officially 256GB of ram (512GB unofficially)
supports Intel Xeon v3 and v4 cpus, a lot of them 2
may be made compatible with standard ATX PUSs using an adapter 3
the front panel connector pinout has been documented 4
supports pci-e bifurcation with the latest bios update 5

Where to find one #

I've searched mine on subito.it, vinted and ebay, in the end I've found one for ~170 euros shipped from germany,
the listing did not specify if the PSU was the 525w or the 700w one and I wanted the latter because of the 12v GPUs rails.
Included there was a Quadro P2000 so it was a fairly good deal.

Memory #

Density and configurations #

The z440 officially supports up to 256GB in 8x 32GB rdimms 6,
unofficially it can go up to 512GB with 64GB rdimms but you'll get an error about unsupported LRDIMMs in the system.

If you populate all the 8 dimm slots, and don't have the fancy ram cooler shroud from HP:

you'll get this error:

Since I had no intention to spend 70+ euros for a shroud I built myself one, see the fans section.

Fans #

Bottom Fan #

The bottom fan has a mostly normal 4pin fan connector, except it has a different key on it, you'll just need to shave the 2 slim guides on the fan connector to adapt it.

Bottom fan mounting #

I've printed the "HP Z440 Fron fan bracket (92mm fan)" from kiril_nedev (https://www.thingiverse.com/thing:4653689)
and the "Fan reduction 120mm to 92mm" from Fixercode (https://www.thingiverse.com/thing:6251457).

Note that the reduction is asymmetric and perfect for our case.

This way I am able to mount a 120mm PWM fan I had lying around in my case, the airflow is excellent and directed to the pci-e slots.

DIY ram cooler #

The RAM cooler uses a spcific connector but the pinout has been found 7:

so I've bought the needed parts on aliexpress:

2x 4pin pwm fan extension (color: "1 in 1"): https://it.aliexpress.com/item/1005006140447407.html
Molex 3.0mm connectors (pack of 10, color: Male H Female T 10Se, 2x3P): https://it.aliexpress.com/item/1005007253431348.html

I've removed the connector from the extension, crimped and soldered the Molex pins:
Ground: black
12V: red
Tach: yellow
PWM: blue

the Tach and PWM pins of the two cables will go to a pin each,
while we'll soler together the Ground and 12V ones since those are common:

To allow the molex connector to fit in the custom HP's plastic receptacle you'll need to remove the locking lever, just shave it off with an xacto kinfe.

Mounting the fans #

I've asked "Snoopmasta" 8 to share with me the awesome fan mount design he did for his z440 and he very kindly sent me the files.

I've printed the parts in esun PLA+ and glued it together with Attack (Cyanoacrliate),
the result is very nice:

sources: #

Proxmox Nvidia Gpu on LXC (pve8-deb12)

2024-12-16T00:00:00+00:00

install nvidia drivers on proxmox 8 (debian 12) #

proxmox 7 is based on debian 12 so we can follow the official steps:
https://wiki.debian.org/NvidiaGraphicsDrivers#Debian_12_.22Bookworm.22$0

the one thing we must remember is to install PVE kernel headers: pve-headers

update pve #

check you already have the correct pve repos,
if needed you cna add it with this (useful also for the lxc container):

add repo key:

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

add repo:

nano /etc/apt/sources.list.d/pve-install-repo.list

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

now we can update everything to the latest kernel:

apt update

apt upgrade

reboot

add repos for non-free nvidia driver #

since nvidia drivers are in the non-free repos let's add them to our repo list:

edit

nano /etc/apt/sources.list

and add "non-free non-free-firmware" where needed:

deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

deb http://security.debian.org/debian-security bookworm-security main
deb-src http://security.debian.org/debian-security bookworm-security main

deb http://deb.debian.org/debian/ bookworm-updates main
deb-src http://deb.debian.org/debian/ bookworm-updates main

then update:

apt update

install headers and nvidia-driver #

now we can install the pve-headers

apt install pve-headers

and install the nvidia driver:

apt install nvidia-driver

at this point we must reboot:

reboot

debug #

if nvidia-smi won't start might be to errors in the driver installation,
if you have any doubt remove everything from nvidia, fix the headers and reinstall

apt remove nvidia*

check driver installation #

run nvidia-smi to check if everything is correctly installed:

nvidia-smi

output:

13:12:42_root@machine:[~]:#nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P400                    On  | 00000000:26:00.0 Off |                  N/A |
| 34%   25C    P8              N/A /  N/A |      1MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

container creation #

in proxmox
select your storage for LXC templates and download the debian 12 template

create a container:

for plex you might want to add the folder on a separate mount:
add disk on path: /var/lib/plexmediaserver
30gb

add gpu to container #

on your pve host check that you see someting like: /dev/dri/renderD128
and

grep render /etc/group

render:x:107:

in proxmox gui edit the container, in "Resources" add: "Device Passtrough"
in "Device Path": /dev/dri/renderD128

select advanced

in "GID in CT": "107" (see the grep render before)

fix container permissions for the gpu: #

in /etc/pve/lxc/.conf

add:

# Allow cgroup access
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm

# Pass through device files
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

and start your container

now from your container you should see:

$ ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 254 Dec 22 20:51 /dev/nvidia-modeset
crw-rw-rw- 1 root root 243,   0 Dec 22 20:51 /dev/nvidia-uvm
crw-rw-rw- 1 root root 243,   1 Dec 22 20:51 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Dec 22 20:51 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Dec 22 20:51 /dev/nvidiactl

source: https://theorangeone.net/posts/lxc-nvidia-gpu-passthrough/#ref-2-configure-container$0

install nvidia drivers on the container #

nano /etc/apt/sources.list

deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb-src http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware

deb http://security.debian.org/debian-security bookworm-security main
deb-src http://security.debian.org/debian-security bookworm-security main

deb http://deb.debian.org/debian/ bookworm-updates main
deb-src http://deb.debian.org/debian/ bookworm-updates main

wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

nano /etc/apt/sources.list.d/pve-install-repo.list

deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

apt update

apt upgrade -y

reboot

apt install pve-headers

apt install nvidia-driver

reboot

nvidia-smi

apt install lshw

lshw -c video

root@plex-gpu:~# lshw -c video
  *-display                 
       description: VGA compatible controller
       product: GP107GL [Quadro P400]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:26:00.0
       logical name: fb0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=nvidia latency=0 resolution=800,600
       resources: irq:73 memory:fb000000-fbffffff memory:d0000000-dfffffff memory:e0000000-e1ffffff ioport:f000(size=128) memory:c0000-dffff

tips and tricks: #

mount bind on containers: #

in /etc/pve/lxc/.conf

mp99 /host/folder/source,mp=/container/target

mp99 /host/folder/source,mp=/container/target_readonly,ro=1

sources: #

https://www.geekbitzone.com/posts/2022/proxmox/plex-lxc/install-plex-in-proxmox-lxc/$0

Restoring data from a broken disk

2024-12-09T00:00:00+00:00

example: #

broken disk: /dev/sdb
- was mounted on /mnt/faulty
external usb disk: /dev/sde
- formatted in ext4 and mounted on /mnt/recovery_8TB

what happened: #

the system started giving errors on a non redundant disk:

Device: /dev/sdb [SAT], 1025 Currently unreadable (pending) sectors
Device: /dev/sdb [SAT], 255 Offline uncorrectable sectors
Device: /dev/sdb [SAT], ATA error count increased from 306 to 903

trying to rsync the files to another disk led to errors and fails

data recovery with ddrescue: #

ddrescue can read the faulty disk to an image (or to another disk) while selecting different settings for aggressivity

we want to create a full disk image on the first pass, avoiding add too much stress on an already faulty drive,
so we'll tune our ddrescue settings to reflect that, for example:

ddrescue --idirect --no-scrape /dev/sdx sdx.img sdx.log

you see some options and settings:

--idirect: skip os caches and work directly on the device
--no-scrape: avoids stressing the disk
/dev/sdb: the faulty device
sdb.img: target disk image
sdb.log: targe disk

the sdx.log file is very important beacause it maps the bad parts of the disk, so in a second pass we can void rescanning the whole drive but we can just insist on the faulty parts

in the subsequent passes we can omit "--idirect" since we already have a full disk image and we want to recover as much as we can, even breaking the drive:

ddrescue --idirect -r3 /dev/sdx sdx.img sdx.log

the only difference with the first pass has been removing "--no-scrape" and adding "-r3", which is the number of retries we want to do on a bad sector

actual data recovery: #

umount /dev/sdb

we are working on the external sub drive, at /mnt/recovery_8TB:

first pass:

ddrescue --idirect --no-scrape /dev/sdb sdb.img sdb.log

output:

Current status
     ipos:   60025 MB, non-trimmed:   131072 B,  current rate:    166 MB/s
     opos:   60025 MB, non-scraped:        0 B,  average rate:  48489 kB/s
     ipos:    1781 GB, non-trimmed:        0 B,  current rate:   13312 B/s
     opos:    1781 GB, non-scraped:    1213 kB,  average rate:  72448 kB/s
non-tried:        0 B,  bad-sector:    93184 B,    error rate:     170 B/s
  rescued:    2000 GB,   bad areas:      145,        run time:  7h 39m 25s
pct rescued:   99.99%, read errors:      306,  remaining time:      1m 35s
                              time since last successful read:          0s
Finished

second pass:

ddrescue --idirect -r3 /dev/sdb sdb.img sdb.log

output:

Current status
     ipos:    1781 GB, non-trimmed:        0 B,  current rate:       0 B/s
     opos:    1781 GB, non-scraped:        0 B,  average rate:      79 B/s
non-tried:        0 B,  bad-sector:   524800 B,    error rate:     170 B/s
  rescued:    2000 GB,   bad areas:      182,        run time:  2h 44m 23s
pct rescued:   99.99%, read errors:     3924,  remaining time:         n/a
                              time since last successful read:     30m 35s
Finished

transferring recovered data: #

now we can move the data from the image file we created to another mountpoint,
to do that we need to mount the image file

detect partitions on the image file:

kpartx -av sdb.img

look what your loop device looks like:

root@machine:/mnt/recovery_8TB# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0   1,8T  0 loop  
`-loop0p1 253:0    0   1,8T  0 part

mount the image in ReadOnly mode:

mount -o loop,ro /dev/mapper/loop0p1 /mnt/recovery_8TB/restored_volume

rsync the data somewhere else

rsync -avh --progress /mnt/recovery_8TB/restored_volume/ /mnt/recovery_8TB/recovered_data/

compare data: #

tree -pugsDx -o tree_faulty.txt -a /mnt/faulty/

tree -pugsDx -o tree_restored.txt -a /mnt/recovery_8TB/recovered_data/

diff tree_*

tree:

p: Print the protections for each file.
u: Displays file owner or UID number.
g: Displays file group owner or GID number.
s: Print the size in bytes of each file.
D: Print the date of last modification
x: do not traverse filesystems

sources:

Proxmox LXC Containers

2024-02-11T00:00:00+00:00

lxc containers #

pros:

lightweight
very flexibile config from proxmox interface

cons:

compromise of the container might affect the running host (vms are better isolated)
backup quirks and downtime (see dedicated section)
can see underlying hardware (see dedicated section)

other things that works:

iptables
vpns (see dedicater section)

backup #

while KVM vms use dirty bitmaps to achieve an online backup, the LXC container needs to be suspended:

snapshot: This mode uses the snapshotting facilities of the underlying storage. First, the container will be suspended to ensure data consistency. A temporary snapshot of the container’s volumes will be made and the snapshot content will be archived in a tar file. Finally, the temporary snapshot is deleted again.

sources:

underlying hardware #

the container will be able to see part of the hardware, for example disks.

An "lbslk" will show the host's disks and you can see infos about the disks,
for example we can retrieve the disk serial:

:#cat /sys/block/sda/device/model
SeagateUltrastarIII

:#cat /sys/class/block/sda/device/wwid   
t10.ATA     SeagateUltrastarIII                          AABBCCDDEEFF

this is an unnecessary potential leak of information that needs to be taken into account,
a VM would only see it's disk image.

detecting the container (useful in Ansible) #

we can leverage

systemd-detect-virt

to understand where our os is running,
the command will output the different technologies if ran without any option, for example:

"none" : running on baremetal
"kvm" : runnning in a vm (on Proxmox)
"lxc" : running in a container (on Proxmox)

See here for the full list:

But if we run the command with the --container option:

systemd-detect-virt --container

the output will be:

"none" : if we are running on ANYTHING else than a container
$container : if we are running insiede a container (so in my case would be "lxc")

This is very useful in Ansible where I want to skip the sysctl tasks since those are not valid for a container:

# playbook.yml:
---
- name: "container detection"
  hosts: localhost
  connection: local
  
  tasks:
    - name: "Register if we are running on anything else than a container (none) or in a container"
      command: systemd-detect-virt --container
      register: systemd_detect_virt

    - name: "Set swappiness to zero in sysctl.conf"
      sysctl:
        name: vm.swappiness
        value: '1'
        state: present
        reload: yes
        sysctl_file: /etc/sysctl.conf
      when: systemd_detect_virt.stdout == "none"

the task will be executed only if the command output is "none", thus we are not inside a container.

tun devices #

to configure a VPN that uses /dev/tun devices an additional configuration is needed:

edit your LXC configfile, for example for container 1001: /etc/pve/lxc/1001.conf

and add:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

sources:

Nginx edit content with ngx_http_sub_module

2024-01-16T00:00:00+00:00

why #

I want to arbitrarly edit the content of a virtual host I am reverse-proxying

prereq #

nginx must be built with "--with-http_sub_module",
to check it already is execute:

nginx -V

example response:

nginx version: nginx/6.42.0
built with OpenSSL 6.4.2g  49 Dec 2048
TLS SNI support enabled
configure arguments: [...CUT...] --with-http_sub_module

what we want is "--with-http_sub_module"

how #

these sub_filter rules will rewrite all the text "banana" to "mango" in your pages

    server {
        listen 443;
        server_name www.site.org;
         ssl on;
         ssl_certificate fullchain.pem;
         ssl_certificate_key privkey.pem;
         ssl_session_cache shared:SSL:10m;
         access_log    /var/log/nginx/www.site.org_access.log;
         error_log     /var/log/nginx/www.site.org_error.log;
         location / {
            proxy_pass http://192.168.1.10/;
            proxy_set_header    Host        $host;
            proxy_set_header    X-Real-IP   $remote_addr;
            #
            sub_filter_once off;
            sub_filter_types *;
            sub_filter 'banana' 'mango';
        }
    }

sources #

https://samanbaboli.medium.com/modify-html-pages-on-the-fly-using-nginx-2e7a2d069086

Docs in markdown - Joplin (bonus points: to static site)

2024-01-14T00:00:00+00:00

why #

I need to maintain some documentation about home/personal projects, while sharing just the home docs with my house mate

Joplin ticks most of what I want:

[x] a cross platform tool where I can organize my notes
[ ] the notes need to be saved in markdown format for future-proofing
[x] sane method of syncing (read: webdav)
[x] encryption
[x] solid tool not under heavy development (I want to adopt it once and keep it for some time)

Bonus points:

[x] the notes need to be easily exportable
[x] the export should be in a format that allows to share them in a static html site

and the winner is.. #

Joplin seems to cover most of what I want, the notes are not in .md format but I can export the with an automation.

What I don't like:
the noted are in markdown format inside the app but everything is saved on a sqlite database.

There's an app for every major platform, and also a cli for headless systems or unattended operations.

runner ups #

Logseq seemed to be the what I wanted but wanted to sync in ways I would not want to support (file sync on every device),
since I also want to support iOS devices a webdav sync is the way to go.

Joplin #

Installation #

Apps #

very simple:
install the app and configure it to sync via webdav

CLI #

install node and the install Joplin:

npm install -g joplin

run joplin

your config files are in ~/.config/joplin

source:

https://joplinapp.org/help/apps/terminal/

WebDav Sync #

Apps #

set it up in the apps

CLI #

enter joplin and use the interactive console via ":",
then run these commands

:config sync.target 6
:config sync.6.path https://example.com/something/webdav/Joplin
:config sync.6.username YOUR_USERNAME
:config sync.6.password YOUR_PASSWORD

then run the sync command:

:sync

E2EE #

(end 2 end encryption)

Apps #

on the first client setup the E2EE and resync,
on the !first clients then sync the items, you will be asked for the master pasword you set up on the first client.

CLI #

run

:e2ee decrypt

you will be asked the master password

afeter every sync you'll need to decrypt your notes:

joplin sync
joplin e2ee decrypt

otherwise when you export your notes you'll receive this error:

This item is currently encrypted: note "test01" (aaa....fff) and was not exported. You may wait for it to be decrypted and try again.

other useful resources #

plugins: https://github.com/joplin/plugins/blob/master/README.md#plugins

Joplin CLI #

useful commands: #

list all the notebooks: #

joplin ls -l /

aaaaa 5 14/01/2024 16:10 Welcome!   
bbbbb   14/01/2024 21:48 Home

headless sync: #

joplin sync

Synchronisation target:  (6)
Starting synchronisation...
Fetched items: 1/1.
Downloading resources...
Fetched items: 1/1. Completed: 14/01/2024 21:31 (3s)

Joplin to Mkdocs #

I want to be able to export the notebook "Home" and transform it to a static html site,
since we are exporting markdown files we can leverage the power and simplicity of Mkdocs to achieve what we want.

Mkdocs #

install mkdocs and create a new project:

pip install mkdocs
mkdocs new mkdocs-prj-name

a new folder called "mkdocs-prj-name" will be created, with this structure:

mkdocs-prj-name
|-- docs
|   `-- index.md
`-- mkdocs.yml

then we can place a directory structure with md files in the folder "docs" and everything will be built

to serve the mkdocs site while working on it or for debugging purposes we can execute this in the project folder:

mkdocs serve -t readthedocs

our site will be accessible locally on http://127.0.0.1:8000/

if you just want to build the site use:

mkdocs build -t readthedocs

your built files will be in the /site folder in the mkdocs workdir:

.
|-- docs
|   `-- index.md
|-- mkdocs.yml
`-- site
    |-- 404.html
    |-- css
    |   |-- fonts
    |   |   |-- Roboto-Slab-Bold.woff
    |   |   |-- Roboto-Slab-Bold.woff2
    |   |   |-- Roboto-Slab-Regular.woff
    |   |   |-- Roboto-Slab-Regular.woff2
    |   |   |-- fontawesome-webfont.eot
    |   |   |-- fontawesome-webfont.svg
    |   |   |-- fontawesome-webfont.ttf
    |   |   |-- fontawesome-webfont.woff
    |   |   |-- fontawesome-webfont.woff2
    |   |   |-- lato-bold-italic.woff
    |   |   |-- lato-bold-italic.woff2
    |   |   |-- lato-bold.woff
    |   |   |-- lato-bold.woff2
    |   |   |-- lato-normal-italic.woff
    |   |   |-- lato-normal-italic.woff2
    |   |   |-- lato-normal.woff
    |   |   `-- lato-normal.woff2
    |   |-- theme.css
    |   `-- theme_extra.css
    |-- img
    |   `-- favicon.ico
    |-- index.html
    |-- js
    |   |-- html5shiv.min.js
    |   |-- jquery-3.6.0.min.js
    |   |-- theme.js
    |   `-- theme_extra.js
    |-- search
    |   |-- lunr.js
    |   |-- main.js
    |   |-- search_index.json
    |   `-- worker.js
    |-- search.html
    |-- sitemap.xml
    `-- sitemap.xml.gz

source: https://www.mkdocs.org/getting-started/

themes #

I've found I like the "redthedocs" theme more than the "mkdocs" one

this is mostly due to the fact that we have a list of the Notebooks and Notes on the left instead of in chaotic and nested menus in the top bar

markupsafe errors #

While building, if you receive an odd error about markupsafe like:

    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.9/dist-packages/markupsafe/__init__.py)

downgrade markupsafe to 2.0.1:

pip install markupsafe==2.0.1 --force

source: https://stackoverflow.com/questions/72191560/importerror-cannot-import-name-soft-unicode-from-markupsafe

Joplin CLI operations #

to export one notebook we'll need to sync, decrypt and the export:

joplin sync
joplin e2ee decrypt
joplin export --notebook "Home" --format md_frontmatter ~/mkdocs-prj-name/docs/Joplin/

keeping it together #

I chose to export Joplin's files in a specific folder inside mkdocs's "docs" folder,
due to the fact that if I execute multiple exports I'll have duplicate files, so I want to clean said folder before exporting.

after the export the mkdocs folder structure will look like this:

mkdocs-prj-name
|-- docs
|   |-- Joplin
|   |   |-- Home
|   |   |   |-- Electricty
|   |   |   |   `-- wiring.md
|   |   |   `-- Water
|   |   |       `-- water_meter.md
|   `-- index.md
`-- mkdocs.yml

NB: the folder structure will be kept, in the mkdocs site menus you'll find the structure:

source: https://discourse.joplinapp.org/t/how-can-i-export-certain-ntoebooks-to-a-certain-system-folder-on-using-the-cli/25362

the script #

#!/usr/bin/env bash

ssh_key="/home/user/.ssh/id_rsa"
folder_base="/data/joplin-to-mkdocs/mkdocs_workdir"
joplin_dest="/docs/Joplin"
rsync_source="/site/"
rsync_user="rsync_user"
rsync_host="192.168.0.10"
rsync_dest="/var/www/html/your-static-site-vhost/"
logfile="/data/joplin-to-mkdocs/autoupdate_script.log"

#wake up ssh:
eval `ssh-agent`
ssh-add $ssh_key

rm -rf $folder_base$joplin_dest

joplin sync
joplin e2ee decrypt

joplin export --format md_frontmatter $folder_base$joplin_dest
# to export just one notebook use: --notebook "NotebookName"

#this substitutes one newline with to newlines in every .md file, it's used for better mkdocs compatibility
find $folder_base$joplin_dest -name "*.md" -exec gawk -i inplace 'BEGIN{RS="\n" ; ORS="\n\n";}; { print }' {} \;

cd $folder_base
mkdocs build -t readthedocs

rsync -avhz --progress $folder_base$rsync_source -e "ssh -i $ssh_key" $rsync_user@$rsync_host:$rsync_dest

rm -rf $folder_base$joplin_dest


echo "- - -" | tee -a $logfile

eval "$(ssh-agent -k)"

scheduling #

I do it via cron:

*/5 * * * * user bash /data/joplin-to-mkdocs/autoupdate_script.sh > /dev/null

workflow #

issues #

formatting #

what I see correctly in Joplin is displayed differently on mkdocs built site, for example the spacings between lines, take this example:

banana
mango

in Joplin a single new line is enough to have the text on different lines:

banana
mango

whilst in mkdocs will be displayed spaced on the same line:

banana mango

this issue should have been patched with the find and aws line in the script,
it seems to persist in lists where the indentation is not respected.

for what I've seen the rest of the formatting is ok

retrieve data from FindMy network and send it somewhere else

2023-11-12T00:00:00+00:00

note: this is still a work in progress

why #

we want to programatically retrieve findmy items data and send it somewhere else

the goal is to ingest that data in traccar

notes #

I am running ios 17.x on an old iphone and macos 14 Sonoma in a vm on proxmox (I wrote an article about that)
this vm has only 2 cpu cores and 4gb of ram (of which only 2 are used right now)
The tests are run with a dedicated icloud account
I am sharing some airtags from a second icloud account
You cannot re-share the tags that have been shared with you

phantom item: #

i've shared a tag, then removed and re-shared before I could accept it, on macos I found 2 invites for the same tag and accepted both. At this point I could only remove the "alive" one since the first one, that have been unshared) cannot be managed or deleted.

phantom notifications #

on the iphone I see the notifications for the invites but don't see the invites inside the section "items" in "findmy"

partial data before accepting shared item #

I've shared a second tag and even before I could accept the invite, I've seen the item data in Items.data

the data is without sensitive data until you accept the tag sharing (see more in "how to read the data in the csv")

airtags #

there's a limit of 16 AirTags per Apple ID

show me the data #

the items (airtags and compatible devices) data is in:
~/Library/Caches/com.apple.findmy.fmipcore/Items.data

converting data to csv #

for our tests we'll use airtag alex script: https://github.com/icepick3000/AirtagAlex
it converts the items data to csv

how to read the data in the csv #

example of the yet to be accepted shared tag:

2023-11-12  16:00:00,"Tag01","Not Available","b777",77777,77,3,"2.0.0",0,null,null,null,null,0,0,0,0,"null","null","null","","","","","","","","","",""

example of the accepted tag (so all data is there)
:

2023-11-12  16:01:00,"Tag01","283KHVFAHSF832","b777",77777,77,3,"2.0.0",4,"crowdsourced",44.113763000000000,12.575151000000000,1699801200000,-1,42.000000000000000,0,-1,"false","true","true","Via Fasulla 0","0","IT","","Lombardia","Via Fasulla","Bergamo","Italy","",""

(the data has been altered for privacy)

use case #

a headless vm with macos Sonoma (which supports items=airtag sharing),
that receives shared airtags and sends the data somewhere else

this way we can leverage the power of FindMy network but still using the tools we like,
we can see this as a "compatibility layer"

what is working/thoughts #

having only macOS online seems to be the way to go, the device sharing arrive very slowly but arrive
having also the icloud account logged on an iphone does not seem to give any added value
- on iOS I see the "phantom notification" of a new item but nowhere to accept it, where on macos at least I can accept the invite
having the macOS user not logged in does not seem to impact the update of the devices

important #

it seems that for having the items update their position you need to have macos logged in and findmy open,
the screen can be locked.

traccar script #

this is a modified version of AirtagAlex's script,
this writes data to the csv and also sends it to traccar

#!/bin/bash

ITEMS_FILE="./Items.data"
CSV_FILE="./Airtags.csv"
CSV_HEADER="datetime,name,serialnumber,producttype,productindentifier,vendoridentifier,antennapower,systemversion,batterystatus,locationpositiontype,locationlatitude,locationlongitude,locationtimestamp,locationverticalaccuracy,locationhorizontalaccuracy,locationfloorlevel,locationaltitude,locationisinaccurate,locationisold,locationfinished,addresslabel,addressstreetaddress,addresscountrycode,addressstatecode,addressadministrativearea,addressstreetname,addresslocality,addresscountry,addressareaofinteresta,addressareaofinterestb"

### VARS for traccar:
traccar_url="http://your.traccar.url:5055"

copy_items_data() {
	echo "Creating a copy of Items.data to prevent potential file corruption"
	if ! cp -p ~/Library/Caches/com.apple.findmy.fmipcore/Items.data "$ITEMS_FILE"; then
	    echo "Failed to copy Items.data file. Please ensure Terminal has 'Full Disk Access' in the 'Privacy & Security' section in macOS Preferences" >&2
	    exit 1
	fi
}

create_csv_file() {
	echo "Checking if $CSV_FILE exists"
	if [ ! -f "$CSV_FILE" ]; then
	    echo "$CSV_FILE does not exist, creating one"
	    if ! echo "$CSV_HEADER" >> "$CSV_FILE"; then
	        echo "Failed to create $CSV_FILE. Please ensure the destination directory is writable." >&2
	        exit 1
	    fi
	fi
}

while true; do
	copy_items_data
	create_csv_file

	echo "Checking number of Airtags to process"
	airtagsnumber=$(jq ".[].serialNumber" "$ITEMS_FILE" | wc -l)
	echo "Number of Airtags to process: $airtagsnumber"
	airtagsnumber=$((airtagsnumber-1))

	for j in $(seq 0 "$airtagsnumber"); do
	echo "Processing airtag number $j"

	datetime=$(date +"%Y-%m-%d  %T")

	serialnumber=$(jq ".[$j].serialNumber" "$ITEMS_FILE")
	name=$(jq ".[$j].name" "$ITEMS_FILE")
	producttype=$(jq ".[$j].productType.type" "$ITEMS_FILE")
	productindentifier=$(jq ".[$j].productType.productInformation.productIdentifier" "$ITEMS_FILE")
	vendoridentifier=$(jq ".[$j].productType.productInformation.vendorIdentifier" "$ITEMS_FILE")
	antennapower=$(jq ".[$j].productType.productInformation.antennaPower" "$ITEMS_FILE")
	systemversion=$(jq ".[$j].systemVersion" "$ITEMS_FILE")
	batterystatus=$(jq ".[$j].batteryStatus" "$ITEMS_FILE")
	locationpositiontype=$(jq ".[$j].location.positionType" "$ITEMS_FILE")
	locationlatitude=$(jq ".[$j].location.latitude" "$ITEMS_FILE")
	locationlongitude=$(jq ".[$j].location.longitude" "$ITEMS_FILE")
	locationtimestamp=$(jq ".[$j].location.timeStamp" "$ITEMS_FILE")
	locationverticalaccuracy=$(jq ".[$j].location.verticalAccuracy // 0" "$ITEMS_FILE")
	locationhorizontalaccuracy=$(jq ".[$j].location.horizontalAccuracy // 0" "$ITEMS_FILE")
	locationfloorlevel=$(jq ".[$j].location.floorlevel // 0" "$ITEMS_FILE")
	locationaltitude=$(jq ".[$j].location.altitude // 0" "$ITEMS_FILE")
	locationisinaccurate=$(jq ".[$j].location.isInaccurate" "$ITEMS_FILE" | awk '{ print "\""$0"\"" }')
	locationisold=$(jq ".[$j].location.isOld" "$ITEMS_FILE" | awk '{ print "\""$0"\"" }' )
	locationfinished=$(jq ".[$j].location.locationFinished" "$ITEMS_FILE" | awk '{ print "\""$0"\"" }' )
	addresslabel=$(jq ".[$j].address.label // \"\"" "$ITEMS_FILE")
	addressstreetaddress=$(jq ".[$j].address.streetAddress // \"\"" "$ITEMS_FILE")
	addresscountrycode=$(jq ".[$j].address.countryCode // \"\"" "$ITEMS_FILE")
	addressstatecode=$(jq ".[$j].address.stateCode // \"\"" "$ITEMS_FILE")
	addressadministrativearea=$(jq ".[$j].address.administrativeArea // \"\"" "$ITEMS_FILE")
	addressstreetname=$(jq ".[$j].address.streetName // \"\"" "$ITEMS_FILE")
	addresslocality=$(jq ".[$j].address.locality // \"\"" "$ITEMS_FILE")
	addresscountry=$(jq ".[$j].address.country // \"\"" "$ITEMS_FILE")
	addressareaofinteresta=$(jq ".[$j].address.areaOfInterest[0] // \"\"" "$ITEMS_FILE")
	addressareaofinterestb=$(jq ".[$j].address.areaOfInterest[1] // \"\"" "$ITEMS_FILE")

	echo "Writing data to $CSV_FILE"
	echo "$datetime","$name","$serialnumber","$producttype","$productindentifier","$vendoridentifier","$antennapower","$systemversion","$batterystatus","$locationpositiontype","$locationlatitude","$locationlongitude","$locationtimestamp","$locationverticalaccuracy","$locationhorizontalaccuracy","$locationfloorlevel","$locationaltitude","$locationisinaccurate","$locationisold","$locationfinished","$addresslabel","$addressstreetaddress","$addresscountrycode","$addressstatecode","$addressadministrativearea","$addressstreetname","$addresslocality","$addresscountry","$addressareaofinteresta","$addressareaofinterestb" >> "$CSV_FILE"


        ### block to write data into traccar:
        #
        serialnumber=`echo $serialnumber | sed 's/\"//g'`
        ### removed due to updated code: tracname=`cat ~/Desktop/Airtags/Items.data | jq .[$j].name | sed 's!"!!g'`
        ### batterystatus=`cat ~/Desktop/Airtags/Items.data | jq .[$j].batteryStatus`
        akku=$((batterystatus * 100))
        # send data to traccar:
	wget --spider $traccar_url/?id=$serialnumber\&lat=$locationlatitude\&lon=$locationlongitude\&speed=0\&user=Airtag\&batteryLevel=$akku\&accuracy=$locationhorizontalaccuracy\&timestamp=$locationtimestamp\&serialnumber=$serialnumber\&lastupdate=$locationtimestamp
        #
        echo "Sleep for 1 second between Airtags"
        sleep 1
        #
        ### END block to write data into traccar

	done
	echo -e "Checking again in 1 minute...\n"
	sleep 60

done

to check: #

sources: #

install and configure tailascale to start at boot on macos

2023-11-12T00:00:00+00:00

we want tailscale to start at boot of our macos vm and not when a user logins (so we can remotely manage the vm).

install go:

brew update && brew install golang
mkdir -p $HOME/go/{bin,src,pkg}

cat <<EOF >> ~/.zshrc
export GOPATH=$HOME/go
export GOROOT="$(brew --prefix golang)/libexec"
export PATH="$PATH:${GOPATH}/bin:${GOROOT}/bin"
EOF

source $HOME/.zshrc

cat <<EOF >> ~/.bashrc
export GOPATH=$HOME/go
export GOROOT="$(brew --prefix golang)/libexec"
export PATH="$PATH:${GOPATH}/bin:${GOROOT}/bin"
EOF

source $HOME/.bashrc

source: https://jimkang.medium.com/install-go-on-mac-with-homebrew-5fa421fc55f5

install tailscaled:

go install tailscale.com/cmd/tailscale{,d}@main

run tailscaled at system boot:

sudo $HOME/go/bin/tailscaled install-system-daemon

tailscale login --authkey=tskey-auth-a08gh083g083208gf08wgef0284ghf08wgf0a288fag30

check it's ok:

tailscale status

output

user@machine ~ % tailscale status
100.123.123.1  this-machine       tailscale-account@ macOS   -
100.321.321.2  another-machine    tailscale-account@ linux   -

# Health check:
#     - This is an unstable (development) version of Tailscale; frequent updates and bugs are likely

NB: as stated, tailscaled is an unstable (development) version of Tailscale.

source: https://github.com/tailscale/tailscale/wiki/Tailscaled-on-macOS

Installing MacOs Ventura on Proxmox

2023-11-12T00:00:00+00:00

why: #

TL;DR:
I want an always-on macos installation without dedicating new hardware to this test.

more:
Some data, like airtags position, is retrievable reliably only from macos,
we've seen that the iCloud APIs ad not reliable or don't expose the data we need.

With a friend we want to test the retrieval of the airtags position from macos, convert the data and send it to traccar, an open source fleet tracking software.

notes: #

This "guide" is a patchwork of various sources I've followed, all of which are cited.
I've wrote this just as a series of notes about what I've done and worked as I wanted to.
This process has been tested with both the Ventura 13 and Sonoma 14 ISOs
The commands you see have been executed on Ventura
The VM will run on proxmox 7.x and on AMD Ryzen series 3000 cpu

hardware acceleration issues #

There are some rendering issues when hardware acceleration is required, this is a known issue and it seems that is solvable only passing a GPU to the vm (not my use case).

Sonoma in particular has some issues on the desktop, the image is not correctly rendered.

These issues are known, there seem to be some workarounds with intel and nvidia gpus (https://dortania.github.io/OpenCore-Post-Install/gpu-patching/intel-patching/#getting-started) but again, it's not my use case.

virtual (and non) hardware #

The vm are running on a cpu Ryzen Series 3000,
the disks are on an nvme drive.

The vm has been installed using 4 cpu cores and 8gb of ram,
now I am using it with 2 cores and 4gb of ram.

create the ISO: #

(I'm running these commands on macos ventura)

check the available software #

?@machine ventura % softwareupdate --list-full-installers

Finding available software
Software Update found the following full installers:
* Title: macOS Sonoma, Version: 14.1.1, Size: 12604952KiB, Build: 23B81, Deferred: NO
* Title: macOS Sonoma, Version: 14.1, Size: 12603757KiB, Build: 23B74, Deferred: NO
* Title: macOS Sonoma, Version: 14.0, Size: 12555162KiB, Build: 23A344, Deferred: NO
* Title: macOS Ventura, Version: 13.6.1, Size: 11662168KiB, Build: 22G313, Deferred: NO
* Title: macOS Ventura, Version: 13.6, Size: 11657005KiB, Build: 22G120, Deferred: NO
* Title: macOS Ventura, Version: 13.5.2, Size: 11655353KiB, Build: 22G91, Deferred: NO
* Title: macOS Ventura, Version: 13.5.1, Size: 11655520KiB, Build: 22G90, Deferred: NO
* Title: macOS Ventura, Version: 13.5, Size: 11654590KiB, Build: 22G74, Deferred: NO
* Title: macOS Ventura, Version: 13.4.1, Size: 11513284KiB, Build: 22F82, Deferred: NO
* Title: macOS Monterey, Version: 12.7.1, Size: 12110635KiB, Build: 21G920, Deferred: NO
* Title: macOS Monterey, Version: 12.7, Size: 12107687KiB, Build: 21G816, Deferred: NO
* Title: macOS Monterey, Version: 12.6.9, Size: 12111110KiB, Build: 21G726, Deferred: NO
* Title: macOS Monterey, Version: 12.6.8, Size: 12119078KiB, Build: 21G725, Deferred: NO
* Title: macOS Monterey, Version: 12.6.7, Size: 12115649KiB, Build: 21G651, Deferred: NO
* Title: macOS Big Sur, Version: 11.7.10, Size: 12125478KiB, Build: 20G1427, Deferred: NO
* Title: macOS Big Sur, Version: 11.7.9, Size: 12125714KiB, Build: 20G1426, Deferred: NO
* Title: macOS Big Sur, Version: 11.7.8, Size: 12120994KiB, Build: 20G1351, Deferred: NO

instructions for macOS 13 Ventura #

install the software installer to create the iso

softwareupdate --fetch-full-installer --full-installer-version 13.6.1

cd ~/Desktop/Proxmox_Ventura/

DISK_SIZE="15361m"
hdiutil create -o ~/Desktop/Proxmox_Ventura/Ventura.cdr -size $DISK_SIZE -layout GPTSPUD -fs HFS+J
hdiutil attach ~/Desktop/Proxmox_Ventura/Ventura.cdr.dmg -noverify -mountpoint /Volumes/install_build
sudo "/Applications/Install macOS Ventura.app/Contents/Resources/createinstallmedia"  --volume /Volumes/install_build --nointeraction --downloadassets
hdiutil detach "/Volumes/Shared Support"
hdiutil detach "/Volumes/Install macOS Ventura"
hdiutil convert Ventura.cdr.dmg -format UDTO -o Ventura.iso
mv Ventura.iso.cdr Ventura.iso
rm Ventura.cdr.dmg

instructions for macOS 14 Sonoma: #

softwareupdate --fetch-full-installer --full-installer-version 14.1.1

cd ~/Desktop/Proxmox_Sonoma/

DISK_SIZE="15361m"
hdiutil create -o ~/Desktop/Proxmox_Sonoma/Sonoma.cdr -size $DISK_SIZE -layout GPTSPUD -fs HFS+J
hdiutil attach ~/Desktop/Proxmox_Sonoma/Sonoma.cdr.dmg -noverify -mountpoint /Volumes/install_build
sudo "/Applications/Install macOS Sonoma.app/Contents/Resources/createinstallmedia" --volume /Volumes/install_build --nointeraction --downloadassets
hdiutil detach "/Volumes/Shared Support"
hdiutil detach "/Volumes/Install macOS Sonoma"
hdiutil convert Sonoma.cdr.dmg -format UDTO -o Sonoma.iso
mv Sonoma.iso.cdr Sonoma.iso
rm Sonoma.cdr.dmg

possible issues: #

note the "-layout GPTSPUD" instead of "-layout SPUD", in the command:

hdiutil create -o ~/Desktop/Proxmox_version/version.cdr -size $DISK_SIZE -layout GPTSPUD -fs HFS+J

it seems that was the change that made the script work,
otherwise I received the error:

Erasing disk: 0%... 10%... 20%... 30%... 100%
Copying essential files...
Copying the macOS RecoveryOS...
Making disk bootable...
Failed to extract AssetData/boot/Firmware/Manifests/InstallerBoot/* from update bundle
The bless of the installer disk failed.

sources: #

preparation #

osk key #

recover the osk key:
it's listed in a court document: https://www.rcfp.org/wp-content/uploads/imported/20120105_202426_apple_sealing.pdf

it seems like apple tried to take it down, arguing it was a trade secret.

I tried to generate it randomly but I only came up with this string,
it seems something is missing, maybe needs to be ROT13, I dunno.

bheuneqjbexolgurfrjbeqfthneqrqcyrnfrqbagfgrny(p)NccyrPbzchgreVap

to check I would have needed to run this command to check but I don't have more time for this topic.

echo "bheuneqjbexolgurfrjbeqfthneqrqcyrnfrqbagfgrny(p)NccyrPbzchgreVap" | tr 'A-Za-z' 'N-ZA-Mn-za-m'

download opencore #

download the latest release of opencore for proxmox: https://github.com/thenickdude/KVM-Opencore/releases

for this guide I've used the v20

decompress the archive and copy the iso, with the macos one, on proxmox

proxmox #

avoiding bootloops #

to avoid bootloops

echo 1 > /sys/module/kvm/parameters/ignore_msrs

echo "options kvm ignore_msrs=Y" >> /etc/modprobe.d/kvm.conf

update-initramfs -k all -u

source: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/ section "Configure Proxmox"

TSC #

check if you have working TSC (time stamp counter):

dmesg | grep -i -e tsc -e clocksource

the output should be:

tsc: Refined TSC clocksource calibration: 3399.998 MHz
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x31024cfe468, max_idle_ns: 440795307017 ns
clocksource: Switched to clocksource tsc

if you see this, then the TSC is not enabled:

TSC synchronization [CPU#0 -> CPU#1]:
Measured 3358870891203288 cycles TSC warp between CPUs, turning off TSC clock.
tsc: Marking TSC unstable due to check_tsc_sync_source failed
clocksource: Switched to clocksource hpet
kvm: SMP vm created on host with unstable TSC; guest TSC will not be reliable

source: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/ section "Requirements"

create the vm #

follow: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/
steps "Create the VM" and "Configure Proxmox"

notes #

you'll need to edit the proxmox vm configfile,
I am on AMD so I've added this:

args: -device isa-applesmc,osk="THE_OSK_KEY" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off -cpu Haswell-noTSX,vendor=GenuineIntel,+invtsc,+hypervisor,kvm=on,vmware-cpuid-freq=on

for intel:

args: -device isa-applesmc,osk="THE_OSK_KEY" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off -cpu host,vendor=GenuineIntel,+invtsc,+hypervisor,kvm=on,vmware-cpuid-freq=on

also remember to change “,media=cdrom” to “,cache=unsafe" otherwise it won't correctly boot.

install macOS Ventura #

upload the opencore ISO and Ventura ISO on Proxmox
follow the steps in: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/
step "Install Ventura"

the main things you want to keep in mind:

remember that when the vm reboots for the installation steps you'll need to manually select "macos installer" in the boot menu, only when the installer has finished all the steps you'll see the same icon but with the disk name you choose when you erased the disk, in my case: "proxmox-sonoma"
in the config wizard at the first boot: do not attach an icloud account right now because we need to fix the serial of the mac before

make the opencore install permament: #

check the disks:

user@proxmox-sonoma ~ % diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *85.9 GB    disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk3         85.7 GB    disk0s2

/dev/disk1 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *16.1 GB    disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS Install macOS Sonoma    15.8 GB    disk1s2

/dev/disk2 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *157.3 MB   disk2
   1:                        EFI EFI                     157.2 MB   disk2s1

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +85.7 GB    disk3
                                 Physical Store disk0s2
   1:                APFS Volume proxmox-sonoma - Data   2.2 GB     disk3s1
   2:                APFS Volume Preboot                 2.0 GB     disk3s2
   3:                APFS Volume Recovery                1.2 GB     disk3s3
   4:                APFS Volume proxmox-sonoma          9.9 GB     disk3s4
   5:              APFS Snapshot com.apple.os.update-... 9.9 GB     disk3s4s1
   6:                APFS Volume VM                      1.1 MB     disk3s6

we want to copy the EFI partition from the Opencore iso to our main EFI partition on the macos disk,
in this case the opencore efi partition is /dev/disk2s1 and the macos disk efi partition is /dev/disk0s1,
so we'll copy them with this command:

sudo dd if=/dev/disk2s1 of=/dev/disk0s1

reboot

source: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/
step "Make the OpenCore install permanent"

activate remote management #

since the proxmox console is a bit slugghish it's better if we work remotely on that vm,
once the install is finished go in "System Settings" --> "General" --> "Sharing"
and activate "Screen Sharing" and "Remote Login"

is VNC

you can define an arbitrary password to access the service

is SSH
we can connect simply via ssh using our username and password:

ssh user@192.168.1.16

next steps: enable ssh-key access and disable password login

set the config.plist parameters #

sources:

https://dortania.github.io/OpenCore-Post-Install/universal/iservices.html

generate the correct serial number and MLB #

retrieve the tool:

git clone --depth 1 https://github.com/acidanthera/OpenCorePkg.git
cd ./OpenCorePkg/Utilities/macserial/
make
chmod +x ./macserial

generate the serials:

./macserial --num 1 --model "iMacPro1,1"

the output will be like:

V832AKJGA831 | B8562969GHIIUT017

the on on the left is our serial, the right one is the Board Serial (MLB)

choose mac address and create the rom address #

we have to choose a mac address which is tied to Apple,
they start with: 00:16:CB
so for example: 00:16:CB:AA:BB:CC

the rom address is the mac address without ":" and all lowercase:
0016cbaabbcc

generate a uuid #

just run "uudigen" and you will have something like:

0DEE5A61-5BDE-431B-803C-3F8C40BCFE0B

mount EFI partition #

our config.plist file is inside the EFI partition which is not mounted by default,
mount our EFI partition and change to the correct partition:

sudo diskutil mount EFI

configure the config.plist #

then we can edit the config.plist in

/Volumes/EFI/EFI/OC/config.plist

add serials #

[..CUT..]
<key>PlatformInfo</key>
        <dict>
                <key>Automatic</key>
                <true/>
                <key>CustomMemory</key>
                <false/>
                <key>Generic</key>
                <dict>
                        [..CUT..]
                        <key>MLB</key>
                        <string>B8562969GHIIUT017</string>
                        <key>ROM</key>
                        <data>0016cbaabbcc</data>
                        <key>SpoofVendor</key>
                        <true/>
                        <key>SystemProductName</key>
                        <string>iMacPro1,1</string>
                        <key>SystemSerialNumber</key>
                        <string>V832AKJGA831</string>
                        <key>SystemUUID</key>
                        <string>0DEE5A61-5BDE-431B-803C-3F8C40BCFE0B</string>
                        [..CUT..]

fix autoboot: #

also set the autoboot, otherwise it will alsways wait foruser input,
in config.plist search Misc -> Boot -> Timeout
and set it to something like "5":

[..CUT..]
        <key>Misc</key>
        <dict>
                <key>BlessOverride</key>
                <array/>
                <key>Boot</key>
                <dict>
                        [..CUT..]
                        <key>Timeout</key>
                        <integer>5</integer> 
                        [..CUT..]

remember to change the mac address on proxmox #

in the hardware section change the macaddress in your vm config

other tips: #

also from: https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/

Disabling SIP (System Integrity Protection)
Upgrading OpenCore

other #

automatic booting #

other than setting a timeout in the config.plist, if you don't encrypt the disk (that would require you to enter a password to unlock the disk) then the system will be reachable via ssh once booted.

what is working: #

icloud login
imessage

sources:

FakeKitten 2, upload anything on amazon photos, now with encryption

2023-11-03T00:00:00+00:00

this was the happy cat we were using as deceit image in the previous version,

now it looks like this:

the updates:

[x] now uses random images
[x] filename is of a raw file (.DNG) so Amazon Photos does not try to fiddle with the image
[x] added encryption
[x] checks for prerequisites (convert and gpg)

you can find the old article here: FakeKitten, or how to trick Amazon Prime Photos to store anything

let's see what changed:

a simple script to upload whatever on Amazon Prime Photos

¯\(ツ)/¯

why: #

Amazon Prime Photos allows you to upload everything as long as it's an image,
giving you unlimited storage (I expect them to do facial recognitioning or other unholy things with your images).

so we can leverage this dynamic to upload any file to resemble an image,
just by appending the original file at the end of a random image we are generating on the fly.

the images will be saved as a raw image file, .DNG extension, so the site won't touch it trying to re-compress the image to save files

I've tried to upload files and managed to get 3.5GB without issues

usage: #

encoding: #

./FakeKitten.sh yourfile.pdf encode

you will obtain a file named:

FakeKitten_143442_yourfile.pdf_3a1863591abdce897987971928512865db_random.DNG

where the file name is composed like this:

[constant]\_[image blocksize]\_[originale filename]\_[sha1sum of the original file]\_[decoy image filename]

at this point you can manually upload it on Amazon Prime Photos

encoding and encrypt your file: #

use encode_enc:

./FakeKitten.sh yourfile.pdf encode_enc

this will use a symmetric password for the encryption with gpg,
you will be asked for said password with a prompt

the different thing with the normal encoding is the filename, which will be:

FakeKittenENC_143442_yourfile.pdf**.gpg**_3a1863591abdce897987971928512865db_random.DNG

decoding: #

decoding will automatically take care of differentiating between an encrypted and non-encrypted original file,
if the file was encrypted you will be asked for the password

./FakeKitten.sh FakeKitten_143442_yourfile.pdf_3a1863591abdce897987971928512865db_random.DNG decode

you will obtain your original file:

yourfile.pdf

The new script: #

#!/usr/bin/env bash

if [ -z "$1" ] || [ -z "$2" ] || [[ $2 != "encode" && $2 != "decode" && $2 != "encode_enc" ]]
	then echo "launch the script with the desired filename and operation"
	echo "./AmazonPrimeWhatever.sh FILENAME OPERATION(encode or decode)"
	exit
fi

# check for prerequisites:
if ! [ -x "$(command -v convert)" ]; then
  echo 'Error: convert (part of the imagemagick suite) is not installed.' >&2
  exit 1
fi

if [ $2 = "encode_enc" ] || [ $(echo $1 | cut -d "_" -f1) = "FakeKittenENC" ]; then
	if ! [ -x "$(command -v gpg)" ]; then
	  echo 'Error: gpg is not installed, it is needed for encryption/decryption' >&2
	  exit 1
	fi
fi

if [ $2 = "encode" ] || [ $2 = "encode_enc" ]; then
        imagejpg="random.jpg"
        image="random.DNG"
        #generate random image:
        mx=320;my=256;head -c "$((3*mx*my))" /dev/urandom | convert -depth 8 -size "${mx}x${my}" RGB:- $imagejpg
        mv $imagejpg $image
        imagesize=$(du -b $image | cut -f1)
	if [ $2 = "encode" ]; then
	        origsha=$(sha1sum $1 | cut -d " " -f1)
	        destimage="FakeKitten_"$imagesize"_"$1"_"$origsha"_"$image
	        mv $image $destimage
	        dd if=$1 bs=1M >> "$destimage"
	        echo "encode completed in $destimage"
	fi
	if [ $2 = "encode_enc" ]; then
		#
		gpg --symmetric --cipher-algo AES256 $1
		#
	        origsha=$(sha1sum $1.gpg | cut -d " " -f1)
	        destimage="FakeKittenENC_"$imagesize"_"$1.gpg"_"$origsha"_"$image
	        mv $image $destimage
	        dd if=$1.gpg bs=1M >> "$destimage"
	        echo "encode completed in $destimage"
	fi
fi


if [ $2 = "decode" ]; then
	imageconst=$(echo $1 | cut -d "_" -f1)
	imagebs=$(echo $1 | cut -d "_" -f2)
	origname=$(echo $1 | cut -d "_" -f3)
	origsha=$(echo $1 | cut -d "_" -f4)
	origimage=$(echo $1 | cut -d "_" -f5)

	dd if=$1 bs=1M skip=$imagebs iflag=skip_bytes > "$origname"

	if [ $imageconst = "FakeKittenENC" ]; then
		#decrypt the file
		orignamenogpg=$(echo "$origname" | sed 's/.gpg//')
		gpg --output $orignamenogpg --decrypt $origname
	fi

	echo "decode completed in $origname, checking file integrity"
	echo $origsha"  "$origname > $origname".sha1"
	shaoutput=$(sha1sum -c $origname".sha1")
	echo $shaoutput
	if [[ $shaoutput != *"OK"* ]]; then
		echo ""
		echo "!!! FAILED SHA VERIFICATION!!! EXITING"
		echo "!!! DELETING ALL CREATED FILES !!!"
		rm $origname
		rm $origname".sha1"
		exit
	fi
        if [ $imageconst = "FakeKittenENC" ]; then
		#remove the file that contains .gpg at the end
                rm $origname
        fi
	rm $origname".sha1"
fi


echo ""
echo "end of my job"

pfsense: install correct realtek driver

2023-05-30T00:00:00+00:00

Scope: #

I have pfsense installed on a pc with realtek NICs,
pfsense does not include the driver for those NICs,
Realtek NICs are unusable without the correct drivers.

Captain Hindsight says: #

"you should have bought an hardware with Intel NICs"

Problems encountered: #

My WAN interface disconnected when dealing with moderate/high traffic,
like a speedtest.

Solution: #

install the realtek drivers

UPDATES 2026-03-13 #

I've upgraded to a newer version and on pfsense 2.7.2 you can install the realtek drivers directly from cli:

pkg search Realtek

realtek-re-kmod-198.00_3       Kernel driver for Realtek PCIe Ethernet Controllers

pkg install realtek-re-kmod-198.00_3

Updating pfSense-core repository catalogue...
Fetching meta.conf:   0%
Fetching packagesite.pkg:   0%
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
Fetching meta.conf:   0%
Fetching packagesite.pkg:   0%
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
	realtek-re-kmod: v196.04_3 -> 198.00_3 [pfSense]

Number of packages to be upgraded: 1

102 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching realtek-re-kmod-198.00_3.pkg: 100%  102 KiB 104.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Upgrading realtek-re-kmod from v196.04_3 to 198.00_3...
[1/1] Extracting realtek-re-kmod-198.00_3: 100%

then reboot and all goes swimmingly

source:

Which version am I running?: #

to understand which version of freebsd you are running look at pfsense homepage under "System Information" --> "Version":

Version	2.6.0-RELEASE (amd64)
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE

we are running "FreeBSD 12.3-STABLE"

Download the correct realtek driver: #

the download link for our version is:
http://pkg.freebsd.org/FreeBSD:12:amd64/release_3/All/realtek-re-kmod-v196.04_3.pkg

but how to retrieve it?

look at the link: "FreeBSD:12:amd64/release_3" means 12.3
"release 3" is our "3"

you can find the correct package version in the correct release repo following this:

go in the main package download repo: https://pkg.freebsd.org
select your arch/version: FreeBSD:12:amd64
(full link: https://pkg.freebsd.org/FreeBSD:12:amd64/)
select your specific release: release_3
(full link: https://pkg.freebsd.org/FreeBSD:12:amd64/release_3/)
since we cannot freely browse /All path, you'll need to download the packagesite.txz and look in the packagesite.yaml for our realtek package by searching "realtek",
or you can use this oneliner:
wget -q -O - https://pkg.freebsd.org/FreeBSD:12:amd64/release_3/packagesite.txz | tar --to-stdout -Jxf - packagesite.yaml | grep realtek | yq -r '.path'
that will output the package path, relative to your position:
"All/realtek-re-kmod-v196.04_3.pkg"
at this point you can combine the two paths:
"https://pkg.freebsd.org/FreeBSD:12:amd64/release_3/" + "All/realtek-re-kmod-v196.04_3.pkg"
finally the complete path: https://pkg.freebsd.org/FreeBSD:12:amd64/release_3/All/realtek-re-kmod-v196.04_3.pkg

Install the driver: #

install the package:

fetch -v http://pkg.freebsd.org/FreeBSD:12:amd64/release_3/All/realtek-re-kmod-v196.04_3.pkg
pkg install realtek-re-kmod-198.00_1.pkg

add the correct lines to load the module on boot,
I've added them to /boot/loader.conf but I've ween suggestions to add the lines to /boot/loader.conf.local

if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"

and then reboot to be safe,
or manually load the module (see Bonus --> Manually load modules) /boot/modules/if_re.ko

Pfsense advanced config: #

in the past I had problems with Hardware Checksum Offload, so I've disabled it:

in PFsense go to: System/Advanced/Networking

and then check "Disable hardware checksum offload"

then reboot

after doing this my system was stable

Bonus: #

Check loaded modules: #

kldstat

and you have to see "if_re.ko":

Id Refs Address                Size Name
 4    1 0xffffffffxxxxxxxx   xxxxxx if_re.ko

Manually load modules: #

kldload module_name.ko

List installed packages: #

pkg info

Remove package: #

pkg remove package_name_version

sources: #

https://www.reddit.com/r/PFSENSE/comments/t872mx/fix_issues_with_realtek_nic_on_pfsense_260/

convert cbz file to epub

2023-04-26T00:00:00+00:00

what is a cbz/cbr file: #

a .cbz file is a compressed archive with files inside

cbz are Comic Book Zip files
cbz are Comic Book Rar files

They usually contain a folder and the filenames of the pages:

comic_title.cbz

comic_title
|-- page1.jpg
|-- page2.jpg
|-- page3.jpg
'-- page4.jpg

cbz to epub conversion: #

conversion: #

to convert a cbz to epub (for max compatibility) I've triedto use Calibre but the results were underwhelming,
using cbr_to_epub (https://github.com/rafalcymerys/cbr_to_epub) is way better:

cbr_to_epub -i "comic_title.cbz" --tile "Comic Title" --author "The Author"

cleanup: #

then you will have 2 files:

comic_title.cbz
comic_title.cbz.epub

then you can bulk rename the converted files:

find . -name "*.epub" -execdir rename 's/.cbz//g' "{}" \;

title: #

The title of the epub is in the metadata and is what the programs reading your epub will cosider for the title.

My filename was something like this: "ComicName IssueNumber ComicTitle.cbz"

so I wanted the filename without the extension to be the title of my epub.

script: #

I am using a script to do that, recursively, in your folder:

#!/usr/bin/env bash

author="Your Author"

for i in *.cbz; do
    title=$(echo $i | cut -d "." -f 1)
    cbr_to_epub -i "$i" --title "$title" --author "$author"
done

find . -name "*.epub" -execdir rename 's/.cbz//g' "{}" \;

for i in *.cbr; do
    title=$(echo $i | cut -d "." -f 1)
    cbr_to_epub -i "$i" --title "$title" --author "$author"
done

find . -name "*.epub" -execdir rename 's/.cbr//g' "{}" \;


#using: https://github.com/rafalcymerys/cbr_to_epub

NB: the script won't take into account some cases, for example when a cbz file is not using zip or when we have errors due to filenames inside the archives
it was too much effort to also consider these cases in the script

issues: #

since a .cbz file is a collection of jpgs in a folder,
and the epub will be built following the filename order,
if we have a broken order, for example:

|-- Comic_Book
|   |-- 000.jpg
|   |-- 1-Title_One-(0).jpg
|   |-- 1-Title_One-(1).jpg
|   |-- 1-Title_One-(10).jpg
|   |-- 1-Title_One-(11).jpg
|   |-- 1-Title_One-(12).jpg
|   |-- 1-Title_One-(2).jpg
|   |-- 1-Title_One-(21).jpg
|   |-- 1-Title_One-....jpg

the file order will be followed even if it's clearly wrong,
in thi case the pages will be mixed up.

000.jpg is the cover.

For this case I've created a script that will fix my issue with filenames,
it will work only on .cbz files since my issue was only on those:

#!/usr/bin/env bash

# 0: WARNING
#
read -p "This process is destructive, I will remove your .cbz files, are you sure you want to continue, do you have a backup? " -n 1 -r
echo   # (optional) move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
    echo "OK, going on.."
else
    echo "Exiting"
    exit
fi

# 1: decompress everything
#
for i in *.cbz; do
    title=$(echo $i | cut -d "." -f 1)
    unzip "$i" -d "$title"
    rm "$i"
done


# 2: fix filenames
#
find . -type f -name "*\(0\)*" -execdir rename 's/\(0\)/\(00\)/g' "{}" \;
find . -type f -name "*\(1\)*" -execdir rename 's/\(1\)/\(01\)/g' "{}" \;
find . -type f -name "*\(2\)*" -execdir rename 's/\(2\)/\(02\)/g' "{}" \;
find . -type f -name "*\(3\)*" -execdir rename 's/\(3\)/\(03\)/g' "{}" \;
find . -type f -name "*\(4\)*" -execdir rename 's/\(4\)/\(04\)/g' "{}" \;
find . -type f -name "*\(5\)*" -execdir rename 's/\(5\)/\(05\)/g' "{}" \;
find . -type f -name "*\(6\)*" -execdir rename 's/\(6\)/\(06\)/g' "{}" \;
find . -type f -name "*\(7\)*" -execdir rename 's/\(7\)/\(07\)/g' "{}" \;
find . -type f -name "*\(8\)*" -execdir rename 's/\(8\)/\(08\)/g' "{}" \;
find . -type f -name "*\(9\)*" -execdir rename 's/\(9\)/\(09\)/g' "{}" \;


# 3: recompress the folders and delete them
#

find . -type d -depth 1 > ./dir_list.txt

cat ./dir_list.txt

while IFS= read -r line; do # Whitespace-safe EXCEPT newlines
    zip -r "$line.cbz" "$line"
    rm -rf "$line"
done < ./dir_list.txt

rm ./dir_list.txt


# 4: convert to epub and fix filename
#
author="Your Author"

for i in *.cbz; do
    title=$(echo $i | cut -d "." -f 1)
    cbr_to_epub -i "$i" --title "$title" --author "$author"
done

find . -name "*.epub" -execdir rename 's/.cbz//g' "{}" \;

problematic cases: #

case 1: broken chars in the archives #

our cbz file has non utf8 chars inside the archive

decompress fip file

unzip "comic_title.cbz" -d "comic_title"

if you are on macOS and have decompress errors due to illegal chars in the filename, es:

checkdir error:  cannot create comic_title
                 Illegal byte sequence
                 unable to process comic_title/page1_text??_text.jpg

use ditto to manage filename format errors:

ditto -V -x -k --sequesterRsrc --rsrc "comic_title.cbz" "comic_title"

then go into the folder and sanitize the filenames with detox:

detox -r -v .

at this point you can re-create your cbz from the fixed folder:

zip -r "comic_title.cbz" "comic_title"

case 2: cbz is not in zip format #

.rar files are not compatible with cbr_to_epub:

decompress your rar archive:

unrar x "comic_title.cbz"

then just recreate the cbz file using zip:

zip -r "comic_title.cbz" "comic_title"

(sorta) monitoring two related systemd services

2023-02-11T00:00:00+00:00

update: systemd supports this feature
see: https://stackoverflow.com/questions/47253020/systemd-stop-dependent-service-when-main-service-crashes

Scope: #

I need to stop service2 when service1 is down.

Why: #

For example in "DIY balancer using DNS" I want to kill bind/named when haproxy is dead so I won't receive any connection that would not be served by haproxy.

Script: #

the script is executed every n minutes by cron,
it checks service1, if it's dead it checks the status of service2 and if needed it kills service2

it also works in the reverse flow:
when service1 is up it checks that service2 is up and running, if not it starts service2

#!/usr/bin/env bash

service1="haproxy.service"
service2="bind.service"
logfile="services_monitor.log"

#check if service1 is running:
if systemctl is-active --quiet $service1 ; then
    echo "$(date +"%Y-%m-%d_%H:%M:%S") -  $service1 UP, which is good"

    #check if service2 is running, if not, let's start it:
    if systemctl is-active --quiet $service2 ; then
      echo "$(date +"%Y-%m-%d_%H:%M:%S") - $service2 UP, all is good"
    else
      echo "$(date +"%Y-%m-%d_%H:%M:%S") - $service1 is UP but $service2 is DOWN, let's start $service2" | tee -a $logfile
      systemctl start $service2
    fi

else
    echo "$(date +"%Y-%m-%d_%H:%M:%S") - $service1 DOWN, which is bad"

    #check if service2 is running, if it is, let's stop it:
    if systemctl is-active --quiet $service2 ; then
      echo "$(date +"%Y-%m-%d_%H:%M:%S") - $service1 is DOWN but $service2 UP, let's stop $service2" | tee -a $logfile
      systemctl stop $service2
    else
      echo "$(date +"%Y-%m-%d_%H:%M:%S") - $service2 is DOWN, all is good"
    fi

fi

Flashing an LSI SAS controller in 2022, AKA "IT-mode" and UEFI

2022-10-16T00:00:00+00:00

Scope: #

I have a controller, LSI SAS9217-4i4e, wit IR firmware and want to switch it to IT firmware.

Why: #

For these SAS controllers there are 2 operating modes:

IR: raid features, your machine won't see the disk directly but will see the raid volumes from the raid controller
IT: passtrough or pure-HBA mode, you machine will see the disks directly and you'll be able to manage them

I want the IT mode because I have some SAS disks I need to format and I can't do it as I like in IR mode.

Ok, but I meant "why you need a SAS controller": #

Well it's not obvious?
In case it isn't, I have some drives lying around and wanted to wipe them.
Also I should be able to play around with some SAS drive enclosure (a box that contains many SAS drives and exits with one or more sas cables).

Steps: #

Download the files from broadcom site:
- from here: https://www.broadcom.com/site-search?q=SAS9217-4i4e
- download: 9217_4i4e_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip
- download: Installer_P20_for_UEFI.zip
With Rufus (on Windows, yes I know..) create a USB key
- not bootable since we are using UEFI
Create the usb key:
- download the EFI Shell from: https://github.com/tianocore/edk/raw/master/Other/Maintained/Application/UefiShell/bin/x64/Shell_Full.efi
- add it to your usb key in the root renamed shell.efi
Copy the SAS controller files, from the archies fcopy these files to the root of your usb drive::
- from Installer_P20_for_UEFI.zip copy:
  - /Installer_P20_for_UEFI/sas2flash_efi_ebc_rel/sas2flash.efi (the flash utility)
- from 9217_4i4e_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows.zip copy:
  - /9217_4i4e_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/sasbios_rel/mptsas2.rom (the controller bios)
  - /9217_4i4e_Package_P20_IR_IT_FW_BIOS_for_MSDOS_Windows/Firmware/HBA_9207_4i4e_IT/9207-4i4e.bin (the controller firmware)

The final folder structure should be like this:

.
├── 9207-4i4e.bin
├── autorun.ico
├── autorun.inf
├── mptsas2.rom
├── sas2flash.efi
├── shell.efi
└── System Volume Information
    └── WPSettings.dat

Prepare the PC:
- disconnect as many disks/peripherals you can from your pc leaving just the controller if possible
- connect your usb drive
Start the EFI shell (tested on my Asus motherboard):
- boot your pc
- go in the BIOS pressing F2 at boot
- in the last page select "Open EFI shell from USB drive"
in the EFI shell:
- execute "map" to see the drivesm they will be "fs0, fs1 etc.."
- identify your correct disk (for example there is written "USB" in the description)
- mount the drive: "mount fs1"
- change folder "fs1:"
- list folder content: "dir"
Flash the controller:
- execute "sas2flash.efi -listall" to list the controller
- execute "sas2flash.efi -o -e 6" to clear the controller bios
- execute "sas2flash.efi -o -f 9207-4i4e.bin -b mptsas2.rom" to flash the new firmware and bios on the controller
- execute "sas2flash.efi -listall" to list the controller
Check your work:
- reboot the computer
- enter in the controller bios (when prompted press "CTRL+C"
- select the controller and you'll see that it's now in IT mode
Congrats!

Sources: #

docker-compose systemd unit file

2022-08-09T00:00:00+00:00

Scope: #

creating a systemd unit file for a docker-compose service

Unit file: #

edit /etc/systemd/system/something.service

Description=Docker Compose service
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/data/docker-compose/service
ExecStart=/usr/local/bin/docker-compose up -d
ExecStop=/usr/local/bin/docker-compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

then start and enable the unit on boot:

systemctl enable --now something.service

SSO with lldap, Authelia and Nginx

2022-07-13T00:00:00+00:00

Scope: #

I wanted to play with a (lightweight) directory service and an identity access management solution to be able to manage the access to some of my sites.

Software: #

lldap: #

lldap
is a lightweight implementation of ldap, lacks some features (like ldaps or clustering) but is ok for my needs right now.
Has many example config file for compatible services.

Authelia: #

Authelia
is a IAM and is layer between your applications/users and ldap.
It has a nice portal and allows you user to configure their devices.

Nginx: #

my reverse proxy of choice, from nginx I can choose which site needs to be authenticated before accessing.

docker-compose: #

I’ve used docker-compose for lldap and Authelia,
pre-reqs:

#create the user for your service:
useradd -u 50000 -U -M -s /bin/false docker_ldap

#create the folder for the local mappings for docker:
mkdir -p /data/lldap /data/authelia

#change the permission on the folders:
chown -R 50000:50000 /data/lldap
chown -R 50000:50000 /data/authelia

the docker-compose.yml:

services:
  lldap:
    image: nitnelave/lldap:stable
    # Change this to the user:group you want.
    user: “50000:50000”
    ports:
      # For LDAP
      - “3890:3890”
      # For the web front-end
      - “17170:17170”
    volumes:
      - “/data/lldap:/data”
    environment:
      - LLDAP_JWT_SECRET=YOUR_SECRET
      - LLDAP_LDAP_USER_PASS=YOUR_PASS
      - LLDAP_LDAP_BASE_DN=dc=example,dc=com

  authelia:
    image: authelia/authelia
    container_name: authelia
    volumes:
      - /data/authelia:/config
    user: “50000:50000”
    ports:
      - “9091:9091”
    restart: unless-stopped
    healthcheck:
      disable: true
    environment:
      - TZ=Europe/Rome

you’ll have to add the Authelia configfile in /data/authelia/configuration.yml
which should look something like this (without comments):

---
theme: auto
jwt_secret: 123107213701371937937
default_redirection_url: https://auth.example.com/
default_2fa_method: “totp”
server:
  host: 0.0.0.0
  port: 9091
  path: “”
  asset_path: /config/assets/
  read_buffer_size: 4096
  write_buffer_size: 4096
  enable_pprof: false
  enable_expvars: false
  disable_healthcheck: false
  tls:
    key: “”
    certificate: “”
    client_certificates: []
  headers:
    csp_template: “”

log:
  level: debug
  format: text
  file_path: /config/authelia.log

telemetry:
  metrics:
    enabled: false
    address: tcp://0.0.0.0:9959

totp:
  disable: false
  issuer: auth.example.com
  algorithm: sha1
  digits: 6
  period: 30
  skew: 1
  secret_size: 32

webauthn:
  disable: false
  timeout: 60s
  display_name: Authelia
  attestation_conveyance_preference: indirect
  user_verification: preferred
  
ntp:
  address: “time.cloudflare.com:123”
  version: 4
  max_desync: 3s
  disable_startup_check: false
  disable_failure: false

authentication_backend:
  password_reset:
    disable: false
  refresh_interval: 1m
  ldap:
    implementation: custom
    url: ldap://lldap:3890
    timeout: 5s
    start_tls: false
    base_dn: dc=example,dc=com
    username_attribute: uid
    additional_users_dn: ou=people
    users_filter: (&({username_attribute}={input})(objectClass=person))
    additional_groups_dn: ou=groups
    groups_filter: (member={dn})
    group_name_attribute: cn
    mail_attribute: mail
    display_name_attribute: displayName
    user: uid=admin,ou=people,dc=example,dc=com
    password: ‘password’

password_policy:
  standard:
    enabled: false
    min_length: 8
    max_length: 0
    require_uppercase: true
    require_lowercase: true
    require_number: true
    require_special: true
  zxcvbn:
    enabled: false
    min_score: 3

access_control:
  default_policy: deny
  rules:
    - domain: ‘public.example.com’
      policy: bypass

    - domain: ‘www.example.com’
      policy: two_factor

session:
  name: authelia_session
  domain: example.com
  same_site: lax
  secret: insecure_session_secret
  expiration: 1h
  inactivity: 5m
  remember_me_duration: 1M

regulation:
  max_retries: 3
  find_time: 2m
  ban_time: 5m

storage:
  encryption_key: 123123123123123123123123123123123123123123123
  local:
    path: /config/db.sqlite3

notifier:
  disable_startup_check: true

  smtp:
    host: smtp.mail.domain
    port: 587
    timeout: 5s
    username: auth.examplecom@mail.domain
    password: password
    sender: “auth.example.com <auth.examplecom@mail.domain>”
    identifier: Auth_examplecom
    subject: “[auth.example.com] {title}”
    startup_check_address: check@test.domain
    disable_require_tls: false
    disable_html_emails: false

    tls:
      skip_verify: false
      minimum_version: TLS1.2
...

Notes:
see “default_policy: deny” this means that if you don’t create a rule for your site (or a rule with a wildcard) then you’ll receive a 403 “Access Denied” after the authentication, ask me how I know :D

lldap has it’s own config (I had to write this from scratch since the container won’t create the correct defaults:

ldap_port = 3890
http_port = 17170

jwt_secret = “123123123123123123”
ldap_base_dn = “dc=example,dc=com”
ldap_user_dn = “admin”
ldap_user_pass = “password”
database_url = “sqlite:///data/users.db?mode=rwc”
key_file = “/data/private_key”

[smtp_options]
enable_password_reset=true
server=“smtp.mail.domain”
port=587
tls_required=true
user=“auth.examplecom@mail.domain”
password=“password”
from=“LLDAP Admin <auth.examplecom@mail.domain>”
reply_to=“Do not reply <noreply@examplecom>”

start the container and access lldap’s web interface on port 17170 with user admin and password what-you-set, then you’ll be able to create other users.

Nginx: #

this is really everything you need to know:
https://www.authelia.com/integration/proxies/nginx/

I’ve put all Authelia’s files in a dedicated config folder and linked them in there,
REMEMBER to create the rules for your site in Authelia’s config.

My nginx config:


#auth.example.com
server {
       listen 80;
       server_name auth.example.com;
       return 301 https://$server_name$request_uri;
       #LOGS:
       access_log  /var/log/nginx/auth.example.com_access.log;
       error_log  /var/log/nginx/auth.example.com_error.log;
}

server {
        listen 443 ssl http2;
        server_name auth.example.com;
        #SSL:
        ssl on;
        ssl_certificate /etc/letsencrypt/live/auth.example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;
        ssl_session_cache shared:SSL:10m;
        #LOGS:
        access_log  /var/log/nginx/auth.example.com_access.log;
        error_log  /var/log/nginx/auth.example.com_error.log;
        location / {
          proxy_pass http://192.168.43.170:9091;
          include /etc/nginx/conf.d/authelia/proxy.conf;
        }
}

#www.example.com
server {
        listen 80;
        server_name www.example.com;
        return 301 $scheme://$server_name$request_uri;
        #LOGS:
        access_log  /var/log/nginx/www.example.com_access.log;
        error_log  /var/log/nginx/www.example.com_error.log;
}

server {
        listen 443 ssl http2;
        server_name www.example.com;
        #SSL:
        ssl on;
        ssl_certificate /etc/letsencrypt/live/www.example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/www.example.com/privkey.pem;
        ssl_session_cache shared:SSL:10m;
        #LOGS:
        access_log  /var/log/nginx/www.example.com_access.log;
        error_log  /var/log/nginx/www.example.com_error.log;
        #
        #authelia:
        include /etc/nginx/conf.d/authelia/authelia-location.conf;
        #
        location / {
          include /etc/nginx/conf.d/authelia/proxy.conf;
          include /etc/nginx/conf.d/authelia/authelia-authrequest.conf;
          root /var/www/example.com;
        }
}

Authelia rules examples: #

in this example inly the users in the group “www_site_users” will be able to access the site www.example.com using the 2 factor authentication, the admin users in the group “www_site_admins” will not be able to access any site of example.com:

    - domain: ‘www.example.com’
      policy: two_factor
      subject: ‘group:www_site_users’

    - domain: ‘*.example.com’
      policy: deny
      subject: ‘group:www_site_admins

Zigbee Sonoff Door (contact) to Leak Sensor

2022-06-22T00:00:00+00:00

UPDATE: #

The sensor has been very unreliable and detected phantom contacts, especially at 3 in the morning, even after applying heat shrink tubing on the contacts.
I’ve ended up taking an Aqara Leak sensor, un screwing the contacts (they are done like this on purpose) and attaching the isolated wire to those. It’s working well.

Scope: #

Conversion of a Sonoff SNZB-04 contact (door/window) sensor to leak (or contact) sensor and code for Homebridge,
possibly retaining the reed switch (magnet) functionality.

Preferably modular so I can attach different "probes" for different use cases.

Why: #

I have a spare contact sensor and need a leak sensor,
I have decided to convert this one instead of buying a new one.

Physical: #

I've soldered 2 wires to the reed swith that would normally close the circuit when a magentic field is applied.
In this way we can close the circuit physically.

I've also modified the case a bit to allow the wires to come out and be soldered to the screw clamps:

this way I can "mount" it in it's final location:

yep, not my proudest soldering work.

Software: #

Using Homebdridge i wanted to define my sensor as a leak sensor insted of a contact sensor.

My config for the contact sensor:

        {
            "accessory": "mqttthing",
            "type": "contactSensor",
            "name": "Sonoff_Door01",
            "url": "mqtt://192.168.0.10:1883",
            "username": "homebridge",
            "password": "password",
            "topics": {
                "getContactSensorState": {
                    "topic": "tele/tasmota_ABCDEF/Sonoff_Door01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0xABCD'].Contact;"
                },
                "getStatusLowBattery": {
                    "topic": "tele/tasmota_ABCDEF/Sonoff_Door01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0xABCD'].BatteryPercentage;"
                }
            },
            "integerValue": "true"
        }

My configuration for the leak sensor:

        {
            "accessory": "mqttthing",
            "type": "leakSensor",
            "name": "Sonoff_Door01_LEAK",
            "url": "mqtt://192.168.0.10:1883",
            "username": "homebridge",
            "password": "password",
            "topics": {
                "getLeakDetected": {
                    "topic": "tele/tasmota_ABCDEF/Sonoff_Door01/SENSOR",
                    "apply": "return !(JSON.parse(message).ZbReceived['0xABCD'].Contact);"
                },
                "getStatusLowBattery": {
                    "topic": "tele/tasmota_ABCDEF/Sonoff_Door01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0xABCD'].BatteryPercentage;"
                }
            },
            "integerValue": "true"
        }

notice the negation "!" in the "getLeakDetected" "apply" section,
this is used because the normal state of the contact sensor is:

"Contact":1 if it's open

"Contact":0 if it's closed

I needed to invert the status otherwise the leak sensor would be always in "leak detection" when there was no contact, thus the use of "!".
This has been done since I haven't found another way to define this in other ways.

The alert arrives promptly and is considered an high-priority alert:

exported iCloud photos have the wrong date

2022-06-02T00:00:00+00:00

Scope: #

One of my friends has a problem:
they use iCloud photos to save/backup photos from the iPhone and want to create an out-of-iCloud backup of the photos on their mac.

The issue is that the photos are exported with consecutive names (es: IMG_1234.JPG) and the files have the creation/modification date of the moment you exported them from Photos to disk.

In this way the real date of the photo is only in Exif data, but we cannot see it directly or sort the photos using the file date.

My friend asked me to create a script to rename the photos with the correct capture date.

Prerequisites: #

brew
- to install exiftool
- install it via: https://brew.sh/
exiftool
- install it via brew:
- https://formulae.brew.sh/formula/exiftool

Script: #

#!/bin/bash

#check if the required tool is installed
if ! exiftool > /dev/null 2>&1 ; then
  echo "exiftool is not installed, exiting"
  exit
fi

#main cycle:
for file in "$@" ; do

  #clean variables every cycle
  filedate=""
  folder=""
  extensione=""
  newname=""

  #differentiate between files that need diferent fields in exiftool:
  if [[ "$file" == *".MOV"* ]]; then
    newname=$(exiftool -d "%Y-%m-%dT%H-%M-%S%z" -MediaCreateDate -S -s "$file")
  else
    newname=$(exiftool -d "%Y-%m-%dT%H-%M-%S%z" -DateTimeOriginal -S -s "$file")
  fi


  if [ ! -z "$newname" ] ; then

    extension=$(echo ${file//*.})
    
    #differentiate if file is in the same folder or another:
    if [[ "$file" == *"/"* ]]; then
      folder=$(echo "$file" | cut -d/ -f1)    
      mv "$file" "$folder/$newname.$extension"    
      #debug:
      #echo "rebuilt: " "$folder/$newname.$extension"
    else
      mv "$file" "$newname.$extension"    
      #debug:
      #echo "rebuilt: " "$newname.$extension"
    fi

  fi


#DEBUG:
    #echo "newname: " "$newname"
    #echo "file: " "$file"
    #echo "extension: " "$extension"
    #echo "folder: " "$folder"
    #echo "- - - - - - - - - - "

done

Usage: #

just run the script against a single file or a folder:

#single file:
./rename.sh IMG_1234.JPG

#folder:
./rename.sh _WORKDIR/*

Unupported files: #

The script ignores filetypes that don't contain (interesting) exif data,
for example images downloaded from the web.

Those files will not be renamed and will retain their original filename.

Notes: #

different files, different exif data: #

I've noticed that .MOV files contain exif data but on another field,
other file types may behave differently.

ISO 8601: #

I've tried to follow ISO 8601 for the dates but MacOS doesn't like ":" in the filenames and in the GUI Finder automatically translates that in "/".

So our filenames will look like:
2022-06-02T13-04-45+0000

Instead of:
2022-06-02T13:04:45+0000

certified for: #

The script has been ran and tested on macOS Big Sur 11.6.6

RFB: request for banners

2022-05-28T00:00:00+00:00

Notice: #

The facts, all banners, commands, and ouputs portrayed in this post are fictitious.
No identification with actual IPs (assigned or retired), banners, scans, and products is intended or should be inferred.

Scope: #

What if we, hypothetically, wanted to retrieve all the banners of ssh server exposed on the internet?
This post analyzes the hypothetical steps I would undertake.

Why: #

Say that we have an honeypot that has hardcoded the ssh banner,
I may want to see which are the most common banners to better blend it amongst real ssh servers.

There's also a simpler way described at the end of the post.

The scan: #

First of all we need to define what we mean by "all ssh servers exposed",
I think we can be more than happy considering only port 22 TCP and not possible different custom ports.

Furthermore I sensed that it would be not wise to scan the whole 0.0.0.0/0 subnet, since there are organizations and people on it's space that are touchy towards unwanted scans,
it would be a clever idea to use an exclusion list to skip those IPs and CIDRs, a suitable list looks like this: exclude.conf (in the same repo of masscan).

The scanner: #

I would use masscan since it has many features and it looks like it's very fast.

The command I think it would look like this:

masscan 0.0.0.0/0 -p22 --banners --source-ip 192.168.0.3 --excludefile exclude.conf -oJ output.json --rate 100000 --connection-timeout 1

Note that for banner scanning we need to provide another ip in the same network, the "--source-ip", see [0]

I would then output everything to a Json file to be able to work the data with Elastic.

For the rate I expect 100000 to be a fair value.

For the connection timeout I expect that 1sec would be more than ok to distinguish between a valid ssh server and something else.

I expect the final file to weight abount 4GB and the scan to take about 10+ hours to complete.

Processing data: #

After grepping only the lines containing the banners from the output file of masscan,
I would totally use Elastic importing the Json and applying the geo-ip filter on the ip field:

I would filter the outputed json data of masscan selecting only the lines with banners:

grep "banner" output.json > output_onlybanners.json

the Logstash config file to import that data would be like:

input {
   file {
      start_position => "beginning"
      path => "output_onlybanners.json"
      sincedb_path => "/dev/null"
      codec =>   json
      type => "json-log"
   }
}

filter {
    if [type] == "json-log" {
        date {
            match => [ "timestamp", "UNIX" ]
        }
    }

    geoip {
      default_database_type => "City"
      source => "ip"
      tag_on_failure => ["geoip-city-failed"]
    }

    geoip {
      default_database_type => "ASN"
      source => "ip"
      tag_on_failure => ["geoip-asn-failed"]
    }

}

output {
  elasticsearch {
       hosts => "http://localhost:9200"
       index => "bannerscan"
  }
}

and I would start the import with this command:

/usr/share/logstash/bin/logstash -f logstash.conf

Reading data: #

Once imported in Elastic we could create our dashboards with Kibana and see which banners are more common so we could select one to use with our honeypot to better blend it.

so our hypothetical list of the most common banners might be something like this:

SSH-2.0-OpenSSH_8.4p1 Debian-5
SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.10
SSH-2.0-dropbear_2017.75

The much simpler way: #

from your machine ask your ssh client the version:

ssh -V 2>&1 >/dev/null | cut -f1 -d','

which results in:

OpenSSH_7.9p1 Debian-10+deb10u2

where the banner taken from a telnet/other connection on the ssh port would be:

SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2

so the difference lies just in that "SSH-2.0-" we can add to the string, resulting in this command:

ssh_banner=$(ssh -V 2>&1 >/dev/null | cut -f1 -d',') ; echo "SSH-2.0-"$ssh_banner

INFO: #

The IPs in the IPv4 space are 4,294,967,296 (2^32) in total, since there are 588,514,304 reserved addresses in the end we have 3,706,452,992 public addresses. [1]

TODO: #

links: #

MapleGrid: DIY distributed SSH honeypot using ELK stack

2022-05-25T00:00:00+00:00

Scope: #

Building an honeypot "network" with a central instance that aggregates the data collected.

Idea: #

The idea is having as many "leaves" as we may so we can sense the "wind" and send everything to a central "trunk" where the info is processed.
This way we can have our little network of ssh honeypots and do something with the data.

the "leaves" will leverage Pshitt made by regit, a python program that simulates an ssh server with user/pass authentication.
Differently from a real ssh server Pshitt stores the data of the brute force access in json format, for example:

{"username": "user", "try": 1, "src_port": 47097, "software_version": "libssh2_1.4.2", "timestamp": "2022-05-25T19:50:00.123456", "src_ip": "123.123.123.123", "mac": "hmac-sha1", "cipher": "aes128-ctr", "password": "password"}

username
try: number of 3 tries allowed before disconnection
src_port
software_version
timestamp
src_ip
mac
cipher
password

Components: #

pshitt #

our ssh honeypot

filebeat #

a lightweight log-shipper to be used on the leaves

logstash #

where we do a little processing of the data and add geo-ip data
I tried using logstash directly on the leaves but it's too resource hungry

elasticsearch #

where we store our data

kibana #

used for exploring data and creating dashboards

Network configuration: #

On the leaves I assume we will have a public ip address, so we can expose:

port 22: pshitt
port 22222: the real ssh server we need to manage the machine

The trunk would not need any exposed port (at least by design) since we should be able to reoute everything via a VPN or protected connection,
in case we cannot do anything like that the port we need to expose:

port 5044: logstash for filebeat

How-To: #

"Trunk" node (ELK stack): #

Install: #

install and configure Elasticsearch on a machine,
preferably with some cores and some GB of ram (I am using 4 cores and 6GB ram),
since I am not very keen on ELK (and I am leveraging this project to learn something), I have used these articles: [0] and [1]

NB: this is a single node "quick and dirty" installation of Elasticsearch and Kibana for a POC.

#install ELK v7.x:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

apt update

apt install elasticsearch kibana logstash filebeat
#filebeat is onyl needed if you wish to host the honeypot on the trunk node


cat <<EOF > /etc/elasticsearch/elasticsearch.yml
cluster.name: maplegrid
node.name: maplenode-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: single-node
node.ingest: true
EOF

cat <<EOF > /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
server.name: "maplegrid-kibana"
elasticsearch.hosts: ["http://localhost:9200"]
EOF

systemctl enable elasticsearch
systemctl enable logstash
systemctl enable kibana
systemctl start elasticsearch
systemctl start logstash
systemctl start kibana

Check if Elasticsearch is ok: #

at this point you should be able to ask Elastic how it feels:

curl -XGET http://localhost:9200/_cluster/health?pretty

expecting an answer like this:

{
  "cluster_name" : "maplegrid",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 15,
  "active_shards" : 15,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 83.33333333333334
}

and you should be able to connect to Kibana on the ip address of the server on port 5601.

Logstash config: #

now we can configure Logstash:

edit /etc/logstash/conf.d/pshitt.conf

input {
  beats {
    port => 5044
  }
}

filter {
    # warn logstash that timestamp is the one to use
    if [type] == "json-log" {
        date {
            match => [ "timestamp", "ISO8601" ]
        }
    }

    geoip {
      default_database_type => "City"
      source => "src_ip"
      tag_on_failure => ["geoip-city-failed"]
    }

    geoip {
      default_database_type => "ASN"
      source => "src_ip"
      tag_on_failure => ["geoip-asn-failed"]
    }

}

output {
  elasticsearch {
       hosts => "http://localhost:9200"
       index => "pshitt"
  }
}

As you can see our input is not a file but we'll listen on port 5044 for our remote filebeat clients.

The "geo-ip" part is needed to add the geo-data info starting from the attacker IP,
the split configuration is needed because the filter can process "City" or "ASN" data but not both at the same time,
so we need to split the configuration. Thanks to leandrojmp [2].

The output is out Elasticsearch node.

the restart Logstash:

systemctl retart logstash

in case you need to debug logstash (I had to), you can read: /var/log/logstash/logstash-plain.log

"Leaf" node (honeypot + logshipper): #

I've create an Ansible playbook to configure everything but we'll obviously see the steps,
it's been test on both Debian 9 and 10.

Ansible Playbook: #

Playbook: maplegrid.yml
Variables file: maplegrid_variables.yml

Add elastic repo and install filebeat: #

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list

apt update

apt install filebeat

change ports: #

in /etc/ssh/sshd_config we'll change the port to 22222
Port 22222

(if you are using it, but you really should) in fail2ban we are gonna change the port of the ssh jail to 22222,
look for the config:

[sshd]
port    = ssh

and change it to:

[sshd]
port    = 22222

!!!!!!!FIX YOU FIREWALL TO ALLOW CONNECTION ON PORT 22222!!!!!!!! (Ansible will do this for you if you are using iptables, otherwise it's your task)

then restart fail2ban and ssh:

systemctl restart fail2ban
systemctl restart ssh

in this moment your connection will remain active,
it's the moment if you can ssh into the machine on port 22222:

ssh user@machine -p 22222

Pshitt: #

install the dependencies of pshitt:

apt install python-pip
pip install python-daemon argparse paramiko

clone the repo of Pshitt in /srv/pshitt:

git clone https://github.com/regit/pshitt /srv/pshitt

create the systemd unitfile we are going to use to start the service,
create the log directory and enable+start pshitt:

cat <<EOF > /etc/systemd/system/pshitt.service
[Unit]
Description=pshitt service
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/srv/pshitt/pshitt.py -p 22 -k /etc/ssh/ssh_host_rsa_key -o /var/log/pshitt/data.json -l /var/log/pshitt/log.log

[Install]
WantedBy=multi-user.target
EOF

mkdir /var/log/pshitt

systemctl enable pshitt
systemctl start pshitt

notice the "-k /etc/ssh/ssh_host_rsa_key",
we are using the keys of our real machine and not the ones shipped with pshitt to avoid being discovered.

at this point if you try to log into your honeypot:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no user@YOUR_PUB_IP -p 22

you should se an entry in /var/log/pshitt/data.json

Filebeat: #

now we can configure filebeat to send the data to logstash (change the strings starting with "YOUR_":

cat <<EOF > /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/pshitt/data.json
  json.keys_under_root: true

processors:
  - drop_fields:
      fields: ["beat", "source", "prospector", "offset", "host", "log", "input", "event", "fileset" ]
  - add_fields:
      target: ''
      fields:
        pshitt_host: YOUR_LEAF_NODE_NAME

output.logstash:
  hosts: ["YOUR_LOGSTASH_IP:YOUR_LOGASTASH_BEAT-PORT"]

#Filebeat service logging:
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640
EOF

systemctl enable filebeat
systemctl start filebeat

at this point if we try another login on the honeypot we should see the data in Kibana.

Kibana: #

log into Kibana and create a new index pattern:

Management --> Kibana --> Index Patterns --> New

Name: pshitt*
Timestamp field: @timestamp

now go to:

Analytics --> Discover

select your index pattern and you should see some data

we can also create dashboards ( Analytics --> Dashboard ),
I've tried to export what I did (it still lacks geo-ip data since I have some issues), but I don't know if the import works:

Kibana Dashboard: kibana_dashboard.njson

My dashboard:

This is the data from 1 hour, we can see the ingestion from different "leaves".

with the field "pshitt_host" we can understand which leaf sent the data,
actually filebeat adds some data so the same info, taken from the vm hostname of the leaf is also found in "agent.hostname" and "agent.name",
I've added it anyway because in the first tests with Logstash I hadn't the additional fields.

"Historical data": #

one of the next steps would be importing the "Historical" data:

This data comes from when I had the predecessor of maplegrid online, which collector more than 5 million unique records in the timespan of 2016-2019 (with some service disruptions in the middle),
at the time the idea was "retrieving" other people's wordlist

Obviously the relevance of the data regarding the IPs is not very much today, but might be interesting to see the statistics of username and passwords, other than testing everything with much more data than now.

On this regard, this is the Logstash configuration to ingest from a local json file:

input {
   file {
      start_position => "beginning"
      path => "/var/log/pshitt/data.json"
      sincedb_path => "/dev/null"
      codec =>   json
      type => "json-log"
   }
}

filter {
    if [type] == "json-log" {
        date {
            match => [ "timestamp", "ISO8601" ]
        }
    }

    geoip {
      default_database_type => "City"
      source => "src_ip"
      tag_on_failure => ["geoip-city-failed"]
    }

    geoip {
      default_database_type => "ASN"
      source => "src_ip"
      tag_on_failure => ["geoip-asn-failed"]
    }

    mutate {
      add_field => { "pshitt_host" => "YOUR_ORIGINAL_HOST" }
    }    
}

output {
  elasticsearch {
       hosts => "http://localhost:9200"
       index => "YOUR_ANOTHER_INDEX"
  }
}

and you can execute logstash with a single config file like this:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/YOUR_CONFIGFILE.conf

TODO: #

[ ] kibana/logstash: fix geo-data
[ ] elasticsearch: understand clustering/data redundancy
[ ] elasticsearch: understand indexes
[ ] enable passwords on Elasticsearch and Kibana
[ ] import "historical" data
[ ] add the ssh version in pshitt:
- pshitt hardcodes "OpenSSH_6.6.1p1 Debian-5" as the ssh version, it should be changed to the real one of the vm (or randomly picked from a list) to avoid the fact that it's an honeypot
- to retrieve the local ssh version: ssh -V 2>&1 >/dev/null | cut -f1 -d','
[ ] add the public ip of the leaf to the data
- this way we can apply geo-ip also to the dest_ip and try to plot that

Notes: #

the "trunk" vm (ELK stack):
- OS: Debian 10
- CPU: 4 core
- RAM: 6gb
- Disk: 40gb
2 "leaves":
- OS: Debian 9
- CPU: 1 core
- RAM: 0.5gb
- Disk: 10gb
2 "leaves":
- OS: Debian 10
- CPU: 1 core
- RAM: 0.5gb
- Disk: 10gb

links: #

DIY balancer using DNS

2022-02-01T00:00:00+00:00

Scope: #

The scope of this project is to provide a pre-tested layer of protection for the publication of a service, via an external machine, ideally a cheap vps without any important data onboard.

This approach allows us to have "higly mobile" services since the won't be directly published to the world.

Requisites: #

no sensitive data on the external machines (so NO ssl certificates)
somewhat redundant
easy-ish to manage
almost-"cattle" approach to the external machines

Pros: #

decoupling the exit (or entrance) point of a service and it's location
almost open participation: if someone in your trust circle wants to contribute, may manage an external machine

Cons: #

managing overhead
more machines
latency (greater the path, greater the latency)
the central power remains whomever manages the DNS
HSTS might break how haproxy reads the SNI request and redirects without terminating the https protocol

Idea: #

layers: #

External: #

in this layer we find the external machines, they are configured with:

haproxy
bind
auto updater script to pull the config changes from a git

the dns records of the services are pointed to those machines,
see the BIND section for more info on the redundancy approach.

VPN: #

this layer provides a layer of anonimity and flexibility between the external machines and the ones with the services onboard.

we made this with a full-mesh network like tinc, which provides a "virtual L2 switch" with alle the machines connected.

the services machines and the external ones connect to these 2 (or more) machines

Services: #

in this layer we find the machines that provide the actual services

what we use: #

haproxy: #

to proxy the connections without terminating the SSL, thus not having any SSL certificate on the external machines, we need to use haproxy to read the SNI and, using ACLs, send the connection to the correct backend.

bind: #

bind provides our poor-fella's redundancy to our machines,
this is done by hosting a bind instance on every external machine and delegating a zone to each of them, for example: balanced.domain.net

domain hosting configuration example:

balanced 300 IN NS machine01.domain.net.
balanced 300 IN NS machine02.domain.net.
balanced 300 IN NS machine03.domain.net.

machine01 300 IN A 100.100.100.1
machine02 300 IN A 100.100.100.2
machine03 300 IN A 100.100.100.3

every machine's bind has a record for a common host inside that zone that is pointed to the machine itself, for example: publish.balanced.domain.net

bind configuration on machine01 example:

$ORIGIN .
; ---Area 1---
$TTL 300      ; 5min

; ---Area 2---
balanced.domain.net       IN      SOA     machine01.balanced.domain.net. root.balanced.domain.net. (
                                  2021100101 ; serial
                                  300      ; refresh (5 min)
                                  300      ; retry (5 min)
                                  600     ; expire (10 min)
                                  300      ; minimum (5 min)
                                );
; ---Area 3---
                IN      NS      machine01.balanced.domain.net.
; ---Area 4---

$ORIGIN balanced.domain.net.
;NOTE: machine01 is the server that solves the names
machine01               300     IN      A        100.100.100.1

;NOTE: here we can define the content of our zone:
publish                 30      IN      A       100.100.100.1
balanced.domain.net.    300     IN      A       100.100.100.1

in this way when we ask the dns for "publish.balanced.domain.net", we are told to go ask the 3 machines to solve the zone "balanced.domain.net", the first machine that we ask to, and is able to solve names, is the one that will deliver our service.

there are no primary and secondary servers, in this configuration all the dns are equal.

example:

we ask the root servers who has "publish.balanced.domain.net"
the root servers say that "domain.net" is managed by the main name-servers for our domain (hoster)
we ask for "publish.balanced.domain.net" at the main name-servers for our domain (hoster)
the hoster's name-servers answer to that the zone "balanced.domain.net" is solved by:
we ask for "publish.balanced.domain.net" to machine02 (choosing randomly or at best by lowest latency [0] [1])
machine02 answers that "balanced.domain.net" is solved by "machine02.balanced.domain.net" (as specified in the zone file)
machine02 finally answers that "publish.balanced.domain.net" is itself, so 100.100.100.2

file sync: #

we want to sync all the configurations on the various servers from a single, maybe shared, point
to do that we'll use a git repo where we'll add the configuration,
a script called by cron will check every 2min if the repo changes and download new haproxy config files and restart the service.

Ansible: #

I've create an Ansible playbook to create the configurations for you,
the playbook will:

install dependencies
install haproxy
generate ssh keypair for git repo and siplay them for you to add them to the repo
checkout git repo to retrieve haproxy config to be used
configure cron to run the sync of the haproxy script every 5min
configure bind
create http folder and start http demo server on port 8000

You'll be able to visit the site on http and see a demo page.

For configuration/running just follow instructions in the README file.

Archive: Ansible_DEMO.tgz

the content of the archive:

Ansible_DEMO/
Ansible_DEMO/README.md
Ansible_DEMO/repo/
Ansible_DEMO/repo/git_hap-config_autoupdate.j2
Ansible_DEMO/repo/index.j2
Ansible_DEMO/repo/bind_zone.j2
Ansible_DEMO/variables.yml
Ansible_DEMO/run_all.sh
Ansible_DEMO/haproxy.cfg_EXAMPLE
Ansible_DEMO/hosts.yml
Ansible_DEMO/final_output.sh
Ansible_DEMO/main.yml

this wants a git repo with your haproxy configfile, just the haproxy.cfg,
I've added an example in the archive.

todo: #

[ ] bind: check TTLs and optimize
[ ] bind: better define failover behaviour
[x] bind: define who-asks-what-to-who in a question-answer way
[X] hap: example configs
[X] filesync: example config
[ ] docs: better images with more info
[X] docs: specify different zones for the external machines to increase redundancy
[ ] docs: list ports required
[x] create ansible playbook to automate the tasks
[ ] general: check if HSTS might break haproxy SNI redirect
[ ] filesync: add gpg repo signing
[ ] filesync: add verify of commit sign when auto updating config file
[x] filesync: add configfile check before copying and restarting service
[x] filesync: add overwrite of local file if differs with the repo's one

links: #

Ejabberd server, selfhosting our chat audio-videocall server

2022-01-23T00:00:00+00:00

Why: #

I've used a Jabber service provided by a friend until now but I wanted to selfhost mine.
In addition we wanted to debug some incompatibilities with iOS clients like Siskin IM with a brand new installation.

Guide I've followed: #

I've followed this post:
https://www.aroundtheglobe.biz/posts/20210819-your_own_xmpp_server_with_ejabberd_on_Debian_11_Bullseye.html

Most of it is perfect and correct,
just the part about Letsencrypt have changed in the last version I think (see dedicated part)

I've also tried to link all the resources I've used in the post and the configfiles.

for the advanced configs I've followed:
https://www.process-one.net/blog/how-to-configure-ejabberd-to-get-100-in-xmpp-compliance-test/

Result: #

[x] chat between devices on the same installation
[x] chat between devices on other instances
[x] audio call on same instance
[x] audio call with users on other instances
[x] video call on same instance
[x] video call with users on other instances
[x] attachments exchange on same instance
[x] attachments exchange with users on other instances
[x] delivery of messages sent when the client was offline
[] sign up page for users (not in my use case)

and 100% score on XMPP Compliance Tester:

Let's start: #

what we are using:

public facing server (I've used Debian 11)
iptables (you'll need the ability to open ports)
domain config (you'll need to have control over your DNS domain config)
ejabberd 21.01-2 from Debian 11 "standard" repos

Install Ejabberd: #

apt install ejabberd

Firewalling: #

We'll need some (a lot) of ports:

tcp/5222 for ejabberd_c2s
tcp/5223 for ejabberd_c2s TLS
tcp/5269 for ejabberd_s2s_in
tcp/5270 for ejabberd_s2s_in TLS
tcp/5280 for ejabberd_http
tcp/5443 for ejabberd_http TLS
tcp/5349 for ejabberd_stun UDP
udp/3478 for ejabberd_stun TCP
tcp/49152:65535 for turn TCP
udp/49152:65535 for turn UDP
tcp/80 to be redirected to tcp/5280

the redirection of port 80 to 5280 is used to allow certbot to generate certificates for our installation, ejabberd will run as a non-privileged user so it won't be able to open ports under 1000.

iptables config:

# Generated by iptables-save v1.8.7 on Sun Jan 23 17:33:37 2022
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [41917:19766826]
:fail2ban-ssh - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -j fail2ban-ssh
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5222 -m comment --comment "ejabberd_c2s plain" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5223 -m comment --comment "ejabberd_c2s tls" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5269 -m comment --comment "ejabberd_s2s_in plain" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5270 -m comment --comment "ejabberd_s2s_in tls" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5443 -m comment --comment "ejabberd_http TLS" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5280 -m comment --comment "ejabberd_http plain" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 5349 -m comment --comment "ejabberd_stun TCP" -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 3478 -m comment --comment "ejabberd_stun UDP" -j ACCEPT
-A INPUT -i eth0 -p udp -m udp -m multiport --dports 49152:65535 -m comment --comment "stun UDP" -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp -m multiport --dports 49152:65535 -m comment --comment "stun TCP" -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -j DROP
-A OUTPUT -o lo -j ACCEPT
-A fail2ban-ssh -j RETURN
COMMIT
# Completed on Sun Jan 23 17:33:37 2022
# Generated by iptables-save v1.8.7 on Sun Jan 23 17:33:37 2022
*nat
:PREROUTING ACCEPT [281:19041]
:INPUT ACCEPT [307:20513]
:OUTPUT ACCEPT [1287:92719]
:POSTROUTING ACCEPT [1287:92719]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 5280
-A OUTPUT -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 5280
COMMIT
# Completed on Sun Jan 23 17:33:37 2022

NET Config: #

I've used a VPS that has a local IP and a Public ip, but the second one is not directly on the network interface of our VM.

Public IP: 111.222.333.444
Private IP: 192.168.1.10

our domain will be: banana.io

(the IPs and domain are fictional)

DNS config: #

we'll need to configure some subdomains:

@
conference
proxy
pubsub
upload

pointing at our server (see: https://www.process-one.net/blog/ejabberd-xmpp-server-useful-configuration-steps/)

and some SRV records for clients and servers, see: https://wiki.xmpp.org/web/SRV_Records

our final DNS config:

@ 300 IN A 111.222.333.444
conference 300 IN A 111.222.333.444
proxy 300 IN A 111.222.333.444
pubsub 300 IN A 111.222.333.444
upload 300 IN A 111.222.333.444

_stun._tcp 300 IN SRV 5 0 3478 conference.banana.io.
_stun._udp 300 IN SRV 5 0 3478 conference.banana.io.
_stuns._tcp 300 IN SRV 5 0 5349 conference.banana.io.
_turn._tcp 300 IN SRV 5 0 3478 conference.banana.io.
_turn._udp 300 IN SRV 5 0 3478 conference.banana.io.
_turns._tcp 300 IN SRV 5 0 5349 conference.banana.io.
_xmpp-client._tcp 300 IN SRV 5 0 5222 conference.banana.io.
_xmpp-server._tcp 300 IN SRV 5 0 5269 conference.banana.io.
_xmpps-client._tcp 300 IN SRV 5 0 5223 conference.banana.io.
_xmpps-server._tcp 300 IN SRV 5 0 5270 conference.banana.io.

I want to try to do the same but on a subdomain, like chat.banana.io instead of banana.io (these are fictional domains)

Create the uploads folder: #

mkdir -p /var/www/upload
chown ejabberd:ejabberd /var/www/upload

Ejabberd config: #

we can find the config in: /etc/ejabberd/ejabberd.yml

###
###              ejabberd configuration file
###
### The parameters used in this configuration file are explained at
###
###       https://docs.ejabberd.im/admin/configuration
###
### The configuration file is written in YAML.
### *******************************************************
### *******           !!! WARNING !!!               *******
### *******     YAML IS INDENTATION SENSITIVE       *******
### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******
### *******************************************************
### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.
###


# loglevel: Verbosity of log files generated by ejabberd
loglevel: info

# rotation: Disable ejabberd's internal log rotation, as the Debian package
# uses logrotate(8).
log_rotate_count: 0

# hosts: Domains served by ejabberd.
# You can define one or several, for example:
# hosts:
#   - "example.net"
#   - "example.com"
#   - "example.org"

hosts:
  - banana.io

certfiles:
  - "/etc/ejabberd/ejabberd.pem"
#  - /etc/letsencrypt/live/localhost/fullchain.pem
#  - /etc/letsencrypt/live/localhost/privkey.pem

# TLS configuration
define_macro:
  'TLS_CIPHERS': "HIGH:!aNULL:!eNULL:!3DES:@STRENGTH"
  'TLS_OPTIONS':
    - "no_sslv3"
    - "no_tlsv1"
    - "no_tlsv1_1"
    - "cipher_server_preference"
    - "no_compression"
    # 'DH_FILE': "/path/to/dhparams.pem"
    # generated with: openssl dhparam -out dhparams.pem 2048

c2s_ciphers: 'TLS_CIPHERS'
c2s_protocol_options: 'TLS_OPTIONS'
s2s_protocol_options: 'TLS_OPTIONS'


#s2s_ciphers: 'TLS_CIPHERS'
###
### SOURCE: https://www.process-one.net/blog/securing-ejabberd-with-tls-encryption/
s2s_use_starttls: required
s2s_dhfile: /etc/ssl/ejabberd/dh2048.pem
s2s_ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"

# c2s_dhfile: 'DH_FILE'
# s2s_dhfile: 'DH_FILE'

listen:
  -
    port: 5222
    ip: 0.0.0.0
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
    #protocol_options: 'TLS_OPTIONS'
    ###
    ### SOURCE: https://www.process-one.net/blog/securing-ejabberd-with-tls-encryption/
    protocol_options:
      - no_sslv2
      - no_sslv3
      - no_tlsv1
      - no_tlsv1_1
    ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
    starttls: true
    tls_compression: false
    dhfile: /etc/ssl/ejabberd/dh2048.pem    
  -
    port: 5223
    ip: 0.0.0.0
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    tls: true
    protocol_options: 'TLS_OPTIONS'
  -
    port: 5269
    ip: 0.0.0.0
    module: ejabberd_s2s_in
    max_stanza_size: 524288
    ###
    ### SOURCE: https://www.process-one.net/blog/securing-ejabberd-with-tls-encryption/
    protocol_options:
      - no_sslv2
      - no_sslv3
      - no_tlsv1
      - no_tlsv1_1
  -
    port: 5443
    ip: 0.0.0.0
    module: ejabberd_http
    tls: true
    protocol_options: 'TLS_OPTIONS'
    request_handlers:
      /api: mod_http_api
      /bosh: mod_bosh
      ## /captcha: ejabberd_captcha
      /upload: mod_http_upload
      /ws: ejabberd_http_ws
  -
    port: 5280
    ip: 0.0.0.0
    module: ejabberd_http
    tls: false
    protocol_options: 'TLS_OPTIONS'
    request_handlers:
      #/admin: ejabberd_web_admin
      /.well-known/acme-challenge: ejabberd_acme
  -
    port: 3478
    #ip: 0.0.0.0
    #transport: udp
    #module: ejabberd_stun
    #use_turn: true
    ### The server's public IPv4 address:
    #turn_ipv4_address: "111.222.333.444"
    ## The server's public IPv6 address:
    # turn_ipv6_address: "2001:db8::3"
    ###
    ### SOURCE:
    ### https://www.process-one.net/blog/how-to-set-up-ejabberd-video-voice-calling/
    transport: udp
    module: ejabberd_stun
    use_turn: true
    turn_min_port: 49152
    turn_max_port: 65535
    ## The server's public IPv4 address:
    turn_ipv4_address: 111.222.333.444
  -
    port: 5349
    ###
    ### SOURCE:
    ### https://www.process-one.net/blog/how-to-set-up-ejabberd-video-voice-calling/
    transport: tcp
    module: ejabberd_stun
    use_turn: true
    tls: true
    turn_min_port: 49152
    turn_max_port: 65535
    #ip returns an error if the ip is not on the interface, so we need to sepcify the private ip here:
    ip: 192.168.1.10
    turn_ipv4_address: 111.222.333.444
  -
    port: 1883
    ip: 0.0.0.0
    module: mod_mqtt
    backlog: 1000


acme:
  ## Staging environment
  #ca_url: https://acme-staging-v02.api.letsencrypt.org/directory
  ## Production environment (the default):
  ca_url: https://acme-v02.api.letsencrypt.org/directory
  auto: true
  #contact: "user@banana.io"
  #ca_url: "https://acme-v02.api.letsencrypt.org"

## Disabling digest-md5 SASL authentication. digest-md5 requires plain-text
## password storage (see auth_password_format option).
disable_sasl_mechanisms:
  - "digest-md5"
  - "X-OAUTH2"

## Store the plain passwords or hashed for SCRAM:
auth_password_format: scram

## Full path to a script that generates the image.
## captcha_cmd: "/usr/share/ejabberd/captcha.sh"

acl:
  admin:
     user:
       - "admin@banana.io"

  local:
    user_regexp: ""
  loopback:
    ip:
      - 127.0.0.0/8

access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: local
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback

api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
    who: all
    what: "*"
  "admin access":
    who:
      access:
        allow:
          - acl: loopback
          - acl: admin
      oauth:
        scope: "ejabberd:admin"
        access:
          allow:
            - acl: loopback
            - acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who:
      ip: 127.0.0.1/8
    what:
      - status
      - connected_users_number

shaper:
  normal:
    rate: 3000
    burst_size: 20000
  fast: 200000

shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast

modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  ## mod_delegation: {}   # for xep0356
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}

  mod_http_upload:
    put_url: https://@HOST@:5443/upload
    docroot: /var/www/upload
    custom_headers:
      "Access-Control-Allow-Origin": "https://@HOST@"
      "Access-Control-Allow-Methods": "GET,HEAD,PUT,OPTIONS"
      "Access-Control-Allow-Headers": "Content-Type"
      #SOURCE: https://www.process-one.net/blog/how-to-configure-ejabberd-to-get-100-in-xmpp-compliance-test/

  mod_last: {}

  mod_mam:
    ## Mnesia is limited to 2GB, better to use an SQL backend
    ## For small servers SQLite is a good fit and is very easy
    ## to configure. Uncomment this when you have SQL configured:
    ## db_type: sql
    assume_mam_usage: true
    default: always

  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      mam: true
      ### SOURCE: https://www.process-one.net/blog/ejabberd-xmpp-server-useful-configuration-steps/
      allow_subscription: true
      persistent: true  
  mod_muc_admin: {}
  mod_offline:
    access_max_user_messages: max_user_offline_messages
  mod_ping: {}
  mod_pres_counter:
    count: 5
    interval: 60
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      "eu.siacs.conversations.axolotl.*":
        access_model: open
      ## Avoid buggy clients to make their bookmarks public
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  ## mod_register:
  ##   ## Only accept registration requests from the "trusted"
  ##   ## network (see access_rules section above).
  ##   ## Think twice before enabling registration from any
  ##   ## address. See the Jabber SPAM Manifesto for details:
  ##   ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
  ##   ip_access: trusted_network
  mod_register:
    ip_access: trusted_network
  mod_roster:
    versioning: true
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_sic: {}
  mod_stream_mgmt:
    resend_on_timeout: if_offline

### SOURCE: https://www.process-one.net/blog/how-to-set-up-ejabberd-video-voice-calling/
  mod_stun_disco:
    credentials_lifetime: 12h
    services:
        -
          host: 111.222.333.444
          port: 3478
          type: stun
          transport: udp
          restricted: false
        -
          host: 111.222.333.444
          port: 3478
          type: turn
          transport: udp
          restricted: true
        -
          host: banana.io
          port: 5349
          type: stuns
          transport: tcp
          restricted: false
        -
          host: banana.io
          port: 5349
          type: turns
          transport: tcp
          restricted: true
  mod_vcard:
    search: false
  mod_vcard_xupdate: {}
  mod_version:
    show_os: false

### Local Variables:
### mode: yaml
### End:
### vim: set filetype=yaml tabstop=8

Generate DH key: #

mkdir /etc/ssl/ejabberd/
openssl dhparam -out /etc/ssl/ejabberd/dh2048.pem 2048

Start the server: #

systemctl start ejabberd.service

About letsencrypt #

NB: I HAVE TO RE-TEST THIS PART:

In one of the latest versions ejabberd supports natively letsencrypt,
just some notes:

From my experience it's better to start with "auto: false" in your ejabberd "acme" configuration,
and issue manually the generation of the certificates:

ejabberdctl request-certificate all

for the webroot give: /etc/ejabberd

when the certificates have been generated then you can switch to "auto: true"

also:
when testing remember to change "ca_url" to the staging url

https://acme-staging-v02.api.letsencrypt.org/directory

otherwise you'll incur in letsencrypt's rate limiting, see: https://letsencrypt.org/docs/rate-limits/

you can check the certificates with this command:

ejabberdctl list-certificates

sample output:

conference.banana.io	/var/lib/ejabberd/acme/live/11111	true
proxy.banana.io		/var/lib/ejabberd/acme/live/22222	true
pubsub.banana.io	/var/lib/ejabberd/acme/live/33333	true
banana.io		/var/lib/ejabberd/acme/live/44444	true
upload.banana.io	/var/lib/ejabberd/acme/live/55555	true

NB: I've yet to test the autorenew but I suppose it should be automatic.

see: https://docs.ejabberd.im/admin/configuration/basic/#acme

NB:
in ejabberd config we are not using ejabberd.pem (I've deleted it),
if you want to remove the entry you have to remove also "certfiles:"

#certfiles:
#  - "/etc/ejabberd/ejabberd.pem"

otherwise you'll get this error:

Failed to start ejabberd application: Invalid value of option certfiles: Expected list, got empty string instead

Useful stuff: #

Create user: #

ejabberdctl register USERNAME banana.io PASSWORD

Logs (are your friends): #

tail -f /var/log/ejabberd/*

Migrating to a new server: #

copy:

/etc/ejabberd
/etc/ssl/ejabberd
/var/lib/ejabberd

to the new machine,
if the Erlang node name is the same (it seems it's based on the hostname of the machine) then you're set, otherwise you'll need to convert the Mnesia database following this guide at "Change Computer Hostname":

https://docs.ejabberd.im/admin/guide/managing/

I've not followed the guide since I've managed to keep the hostnames the same so i've just moved the files and installed ejabberd, and voila', i was set to go.

Clustering: #

Not in my todo list, but here it is:

https://docs.ejabberd.im/admin/guide/clustering/

Sources: #

https://www.aroundtheglobe.biz/posts/20210819-your_own_xmpp_server_with_ejabberd_on_Debian_11_Bullseye.html

Home Automation

2022-01-07T00:00:00+00:00

Index: #

Preface
Platform
Sensors
Homebridge
Home Assitant
Telegraf, Influx and Grafana
Todo
Homebridge examples
- Straight MQTT integration
- Zigbee devices

Preface: #

With the help of a friend, I wanted to try to see how deep was the rabbit's hole of self-made home automation,
this is the full extent of the installation which now has been reduced:

MQTT: is my MQTT server made with Mosquitto
homeassistant: is the Home Assistant installation
homebridge: is the Homebridge isntallation
Tasmotized Devices: are all devices running Tasmota
MQTT Generic Devices: are devices sending/receiving MQTT messages (without Tasmota)
ZBBridge: allows ZigBee devices to speak MQTT
Apple Devices: are an iPhone or iPad
Android/other Devices: are Android/other OS devices that supports Home Assistant
Telegraf: is a service that reads MQTT messages and puts the metrics into InfluxDB
Influxdb and Grafana: is my system to record and plot metrics
Homekit Native Devices: are devices working only with homekit network (like a smart plug)
Homekit Hub: can be an Apple TV, an Homepod or an iPad

Platform: #

After installing and using Home Assistant for a bit, I decided to decouple the MQTT server, which will be the center of my installation, and the rest.

In this way Home Assistant wil be a "satellite" of MQTT like all the other parts.

In the end we have two main ways to interact with the sensors:

via Home Assistant
via Apple Home

Sensors: #

We are using different types of sensors:

Tasmota (Wifi devices)
Homekit Native (Wifi devices)
ZigBee (Zigbee protocol)

Tasmota: #

Tasmota is an open source firmware for ESP devices that allows you to control the flashed device via MQTT, WebUi or http.

We use Tasmota flashing it's firmware on commercial devices or self-made devices, here is a list of supported tasmota devices.

NB: You could flash Tasmota via Tuya-convert on Tuya devices but unfortunately Tuya updated it's devices so no conversion can take place on-the-fly or worst, they use something different than ESP boards so Tasmota is not supported anymore on those devices.

here we'll see the devices I use with Tasmota:

Sonoff Mini: #

An in-wall switch module I used to take over existing single switches and make them remotely controlled,
this retains the functionality of the physical switch that now is connected to the Sonoff Mini.

https://templates.blakadder.com/sonoff_mini.html

Shelly 1: #

An in-wall switch module like Sonoff Mini but I sued it for double deviators.
https://templates.blakadder.com/shelly_1.html

Koogeek W-DEXI: #

A power-monitoring plug

https://templates.blakadder.com/kogeek_W-DEXI.html

Arilux LC03 RGB Module: #

An RGB controller

https://templates.blakadder.com/arilux_SL-LC_03.html

Ikea Vindriktning: #

I've "Tasmotized" this device adding an ESP-8266 inside it.

https://blakadder.com/vindriktning-tasmota/

Node-MCUs #

These are ESP-8266 self-made devices:

I have some of them, one is connected to a relay board to act as a switch and open a door (otherwise opened with a push-button),
another has a DHT22 to act as a thermometer-hygrometer, another has a IR emitter to send IR messages to interact (blindly, so no messages a received back) with IR-controlled appliances.

Sonoff ZBBridge: #

This device is a Sonoff Zigbee Bridge, reflashed with tasmota.

It allows me to integrate ZigBee devices with the rest of the isntallation (see ZigBee chapter).

https://zigbee.blakadder.com/Sonoff_ZBBridge.html

Zigbee: #

Zigbee is a low-power consumption standard for small meshed local network of wireless devices.

Usually, for home automation, you'll need an hub to integrate ZigBee devices in your installation, I've used a ZBBridge reflashed with Tasmota, this allows me to have a zigbee2tasmota integration, my goal however was just having a zigbee-mqtt compatibility layer.

There many vendors that produce devices compatible with my implementation, I found a compatibility Repository here: Blakadder Zigbee Device Compatibility Repository.

The devices are battery powered, but only send a message when there is a change of state, for example a thermomter-hygrometer will send a message only when temp/hum changes.

This allows to have a long battery life (allegedly).

Right Now I am using:

Sonoff SNZB-01 Wireless Switch
Sonoff SNZB-02 Temperature And Humidity Sensor
Sonoff SNZB-03 Motion Sensor
Sonoff SNZB-04 Wireless Door/Window Sensor
Ikea Tradfri E1745 Tradfri Motion Sensor
Ikea Tradfri LED bulb E27 806 lumen, dimmable
Ikea Tradfri E1743 Tradfri ON/OFF Switch
Aqara Smart Plug SP-EUC01

All the Sonoff devices are well made, compact and work very well.
The Ikea ones are a bit disappointing, the motion sensor for example sends just a message when it detects motion and not when it stops detecting, it just resets after 180 seconds,
the ON/OFF switch worked for 1 day and the disappeared and won't talk with the bridge..

ZB_Repeater #

Some of the always-powered devices (not the battery ones) like the Aqara Smart Plug or the IKEA Tradfri bulb, can act as mesh repeater in the ZigBee network, in this way if your Bridge does not cover all your area you can rely on the Repeater.

NB: in my experience I had to re-join the devices to the bridge after adding the repeaters, otherwise they won't mesh with them.

Homekit Native Devices: #

I am using some Meross/Refoss Wifi smart plugs.
They have static DHCP mappings and are denied any network destination via the firewall.

Homebridge: #

Homebridge is a fantastic piece of software that allows you to integrate many non-supported devices in your Homekit installation,
you can use many of the available plugins,
I am using homebridge-mqttthing to use my MQTT devices, some examples: Homebridge examples

Homekit: #

Homekit is Apple devices's native home management software.
Paired with an Hub, which is a device that is always on and connected to your network, it allows you to create automations and use the devices from outsoide of your wifi network.

Home Assistant: #

Does the same as Homekit, in a much more flexible way.

Telegraf, Influx and Grafana: #

or "TIG".
it's what I already use to collect and plot metrics,
so I wanted to add the metrics of the new devices, for example the Sonoff Temp/Hum and Ikea's Vindriktning.
Telegraf has an MQTT Consumer Plugin, the version that is working for me is Telegraf >= 1.21.

These is my /etc/telegraf/telegraf.d/mqtt.conf:

[[inputs.mqtt_consumer]]
  servers = ["tcp://192.168.1.110:1883"] #your mqtt server

  ## Topics that will be subscribed to.
  topics = [
    "tele/tasmota_ZBBridge/ZB_Sonoff_Temp01/SENSOR",              ###Zigbee Temp01
    "tele/tasmota_Vindriktning/SENSOR",                               ###VINDRIKTNING Tasmota
  ]

  qos = 0
  connection_timeout = "30s"
  username = "user"	#influxdb user
  password = "pass"	#influxdb pass
  data_format = "json"


[[processors.regex]]
  [[processors.regex.field_rename]]
    pattern = '(^ZbReceived_)\w+_'
    replacement = "${2}"

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "sensors"

pay particular attention to "processors.regex"
(AFAIK available from Telegraf >= 1.21),
since it's the component that keeps our metric collectioning sane.

Without this block all our measurement would contain the Topic, for example if we receive a message:

tele/tasmota_ZBBridge/ZB_Sonoff_Temp01/SENSOR {"ZbReceived":{"0x1111":{"Device":"0x1111","Name":"ZB_Sonoff_Temp01","Humidity":94.84,"Endpoint":1,"LinkQuality":34}}}

Telegraf would log the metric like this:

ZbReceived_0x1111_Humidity

when we just want "Humidity" because having more than 1 Thermometer would drive us crazy with many different metrics,
after the processor.regex our metric looks like this:

Humidity

This applies particularly to Zigbee devices as you can see from "ZbReceived".

This is the other configfile of telegraf (the main one), /etc/telegraf/telegraf.conf:

[global_tags]

[agent]
  logfile = "/var/log/telegraf/telegraf.log"
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false

the result:

as you can see the metrics from the Zigbee devices are not always updated, on the contrary of Tasmota's ones that are updated on a schedule.

Todo: #

Nodered

Homebridge examples: #

Straight MQTT integration: #

Tasmota WDexi Plug, Sonoff Mini, Shelly 1, : #

        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "topics": {
                "getOn": "stat/tasmota01/POWER",
                "setOn": "cmnd/tasmota01/POWER"
            },
            "type": "lightbulb",
            "name": "Tasmotized-WDEXI",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "onValue": "ON",
            "offValue": "OFF"
        }

NodeMCU Temp and Hum: #

        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "type": "temperatureSensor",
            "name": "Tasmota-NodeMCU-01_Temperature",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getCurrentTemperature": {
                    "topic": "tele/tasmota_Nodemcu01/SENSOR",
                    "apply": "return JSON.parse(message).AM2301.Temperature;"
                }
            }
        },
        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "type": "humiditySensor",
            "name": "Tasmota-NodeMCU-01_Humidity",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getCurrentRelativeHumidity": {
                    "topic": "tele/tasmota_Nodemcu01/SENSOR",
                    "apply": "return JSON.parse(message).AM2301.Humidity;"
                }
            }
        }

RGB Controller: #

NB: it uses the plugin "sonoff-tasmota-mqtt-hsb"

        {
            "plugin_map": {
                "plugin_name": "homebridge-sonoff-tasmota-mqtt-hsb",
                "index": 0
            },
            "accessory": "sonoff-tasmota-mqtt-hsb",
            "name": "STUDIO - Scrivania Led RGB - Tasmota-RGB-01",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "caption": "test_rgb",
            "topics": {
                "setOn": "cmnd/tasmota_RGB/Power",
                "setHsb": "cmnd/tasmota_RGB/HSBColor",
                "status": "stat/tasmota_RGB/RESULT",
                "getHsb": "stat/tasmota_RGB/HSBColor"
            }
        }

Ikea Vindriktning: #

        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "type": "airQualitySensor",
            "name": "Vindriktning",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getAirQuality": {
                    "topic": "tele/tasmota_Vindriktning/SENSOR",
                    "apply": "return (JSON.parse(message).VINDRIKTNING['PM2.5'] > 55 ? 'POOR' : JSON.parse(message).VINDRIKTNING['PM2.5'] > 35 ? 'INFERIOR' : JSON.parse(message).VINDRIKTNING['PM2.5'] > 12 ? 'FAIR' : JSON.parse(message).VINDRIKTNING['PM2.5'] > 4 ? 'GOOD' : JSON.parse(message).VINDRIKTNING['PM2.5'] <= 3 ? 'EXCELLENT' : 'UNKNOWN');"
                },
                "getPM2_5Density": {
                    "topic": "tele/tasmota_Vindriktning/SENSOR",
                    "apply": "return JSON.parse(message).VINDRIKTNING['PM2.5'];"
                }
            },
            "airQualityValues": [
                "UNKNOWN",
                "EXCELLENT",
                "GOOD",
                "FAIR",
                "INFERIOR",
                "POOR"
            ]
        }

ZigBee devices: #

this is a bit trickier since when you send a command via MQTT it also needs to be sent via Zigbee,
so you have to add "ZbSend" command and payload.

Sonoff Motion Sensor: #

        {
            "accessory": "mqttthing",
            "type": "occupancySensor",
            "name": "ZB_Sonoff_Move01",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getOccupancyDetected": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Sonoff_Move01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0x1111'].Occupancy;"
                }
            },
            "integerValue": "true"
        }

Sonoff Button: #

        {
            "accessory": "mqttthing",
            "type": "statelessProgrammableSwitch",
            "name": "ZB_Sonoff_Butt01",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getSwitch": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Sonoff_Butt01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0x2222'].Power;"
                }
            },
            "switchValues": [
                2,
                1,
                0
            ]
        }

Sonoff Temp and Hum sensor: #

        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "type": "temperatureSensor",
            "name": "ZB_Temp01 - TEMP",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getCurrentTemperature": "tele/Tasmota_ZBBridge/ZB_Sonoff_Temp01/SENSOR$.ZbReceived[*].Temperature"
            }
        },
        {
            "accessory": "mqttthing",
            "type": "humiditySensor",
            "name": "ZB_Temp01 - HUM",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getCurrentRelativeHumidity": "tele/Tasmota_ZBBridge/ZB_Sonoff_Temp01/SENSOR$.ZbReceived[*].Humidity"
            }
        }

Sonoff Door sensor: #

        {
            "accessory": "mqttthing",
            "type": "contactSensor",
            "name": "ZB_Sonoff_Door01",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getContactSensorState": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Sonoff_Door01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0x4444'].Contact;"
                }
            },
            "integerValue": "true"
        }

Aqara Plug: #

        {
            "accessory": "mqttthing",
            "type": "outlet",
            "name": "ZB_Aqara_Plug01",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getOn": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Aqara_Plug01/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0x5555'].Power;"
                },
                "setOn": {
                    "topic": "cmnd/Tasmota_ZBBridge/ZB_Aqara_Plug01/ZbSend",
                    "apply": "return '{\"device\":\"0x5555\", \"send\":{\"Power\":' + message + '}}'"
                }
            },
            "integerValue": "true"
        }

Tradfri Dimmerable Bulb: #

        {
            "accessory": "mqttthing",
            "plugin_map": {
                "plugin_name": "homebridge-mqttthing",
                "index": 0
            },
            "type": "lightbulb",
            "name": "ZB_Tradfri_Bulb11",
            "url": "mqtt://192.168.1.110:1883",
            "username": "username",
            "password": "password",
            "topics": {
                "getOn": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Tradfri_Bulb11/SENSOR",
                    "apply": "return JSON.parse(message).ZbReceived['0x6666'].Power;"
                },
                "setOn": {
                    "topic": "cmnd/Tasmota_ZBBridge/ZB_Tradfri_Bulb11/ZbSend",
                    "apply": "return '{\"device\":\"0x6666\",\"send\":{\"Power\":\"' + message + '\"}}'"
                },
                "getBrightness": {
                    "topic": "tele/Tasmota_ZBBridge/ZB_Tradfri_Bulb11/SENSOR",
                    "apply": "return Math.round(JSON.parse(message).ZbReceived['0x6666'].Dimmer / 2.55)"
                },
                "setBrightness": {
                    "topic": "cmnd/Tasmota_ZBBridge/ZB_Tradfri_Bulb11/ZbSend",
                    "apply": "return '{\"device\":\"0x6666\",\"send\":{\"Dimmer\":' + Math.round(message * 2.55) + '}}'"
                }
            },
            "onValue": "1",
            "offValue": "0",
            "integerValue": "true"
        }

What Is A Computer, Tests on a Lent iPad

2022-01-06T00:00:00+00:00

Preface: #

I've been lent an iPad (with usb-c port) and wanted to test the motto "What's a computer?",
so I've tried to use it as a companion device (NB: not my main device which still is and will be a proper computer).

the device has:

11" screen
usb-c port
external keyboard with trackpad (not Apple)
glass screen protector
not an M1 model

Checklist: #

Important features: #

[x] git repos: can pull (also in a local folder), modify and push multiple repos, using Working Copy (20usd) which also has a very nice editor.
[ ] syncthing: there is a closed source client (Mobius Sync), not tried and don't think I will
[x] nextcloud: supported via it's official app, seems to work well
[x] password manager: sorta yes with workaround, see dedicated section
[x] external monitor support: yes, see dedicated section
[x] local shell: yes with iSH, see dedicated section
[x] local port forwarding via ssh: yes in iSH with caveats.
[x] printing: yes with airprint, see dedicated section
[x] VPN: yes/sorta, see Comms and dedicated section

Comms: #

[~] whatsapp: via web version in a browser, no notifications but works well enough for me
[x] telegram: has app with multiple clients supported, so no sweat here
[x] signal: has app and will join as a connected device
[x] jabber: via chatsecure, works but the problem is with the omemo keys which get scrambled and my interlocutors get pissed.
[ ] TINC vpn: not supported, only via cydia
[x] WireGuard VPN: supported and very well via it's app

Misc: #

[ ] Isolate work apps: no multiple user support
[ ] Multiple identities in browser: not supported
[ ] adblock on youtube: not supported
[ ] sorta no adblock in the browser (you can do something in settings)

Devices: #

[~] AC511 Dell Soundbar: supported if connected directly, but not trough USB-C hub
[x] USB-c hub with power delivery to connect everything with one cable
[x] Usb storage devices: pens and disks work, they mustn't be power hungry.

iSH: #

The joy of my eyes: iSH
it's Linux shell environment running locally, using a usermode x86 emulator, based on Alpine.
It's performances are not stellar since it's an emulation, but you have a complete (but limited) working environment.

PROs:

almost complete work environment
SSH port forwarding works well, as long as you keep iSH in the foreground, which with split screen view is quite comfortable and usable, when iSH goes in background after a bit the SSH sessions goes down and with it your port forwarding.
ability to install packages with "apk add"

CONs:

Lack of ALT or ESC button: even on hardware keyboards like the ones in the cases there is no ALT or ESC, but you can work around it with a soft-button in the bar of iSH, hardware keyboard connected via bluetooth however have the ALT key and with the best testeditor (obviously nano) works very well.
NO NETWORK SUPPORT: you can use the network but you cannot modify anything, this means that tinc vpn won't work because it would modify netowkr interfaces, however Wireguard, via it's app, provides everything I need.
slow, but that's a compromise worth doing.

Password Manager: #

I am using Unix Pass as my password manager,
because it's on CLI and because I can use a gpg key to encrypt and decrypt my passwords, I can keep everything in sync with a git repo and it's fully functional.
I've tested with a new keypair and repo, in iSH and there's everything you need to have: git, pass, gpg, It's not very fast but it works.
It has no completion with tab when you go around in the secrets list.

VPN: #

some of my VPNs are Tinc vpn, which unfortuntely is not supported.
However wireguard has a very good app that works well,
I've tried bridging multiple vpns on a vm so I can connect via wireguard but use resources on the Tinc vpn and it works, quite well (you'll exit "natted" to the Tinc vpn, with the Tinc ip of the bridge machine).
Other more commercial VPNs are supported but don't need 'em.

External Monitor: #

Works well via adapter or via USB-c powered HUB,
it won't cover the whole monitor, I am using a 1920x1080 screen and I have 2 black bars to keep the aspect ratio of the iPad monitor,
on almost 47cm of horizontal I have 38cm of image and 9cm of black bars (4.5cm per side).

Printing: #

Works with a workaround:
I've installed an old Single Board Computer with CUPS and enabled an AirPrint server.
I think I followed this guide
Works surprisingly well.

Accessories: #

There's a wide gamut of accessories for these devices,
with a lot of money to spend.
The keyboards with a decent touchpad that support gestures make the iPad a notebook, more surface-y, like device.
The gestures on the trackpad are really nice and ease moving between apps.

Media consumption: #

It was made for media consumption, so we are ok.

Battery life: #

From what I've tried the battery life is very good, confronted with my old Surface Go the iPad can show 1h of Netflix or do 1h of videocall using very little battery,
I think that it would last more than a day.

Conlusions: #

The experience is eerly,
it's quite a capable device that wants to look like a computer but still tries to keep it's distance from being one. (I'm still doubtful why Apple won't run MacOs on the M1 iPad.)

Limited to my (uncommon) use case: this solution has many compromises, particularly security wise:
it can do almost everything I need swiftly, but I have to trust it (or it's apps) with some of my most beloved "keys".

This device looks like the missing link between a PC and a device that can be mobile, that lasts on battery and has a neat interface (more when paired with a trackpad and keyboard),
and it looks like a device that is keeping all the promises the Surface Go made, promises that even on Windows (Linux is clunkier on the Go) are not kept.

It feels like choosing between what it couldn't be and a compromise.

Automating Eleventy Fetching, Building and Publishing

2022-01-04T00:00:00+00:00

Scope: #

Automating the publishing of eleventy static blog

How: #

You will need:

linux vm (I've used Debian 11)
git repo
eleventy docker image (I've used https://github.com/femtopixel/docker-eleventy)
web server or service of your choice to host your static site
script to check git repo and start the build if there are any changes

Pros and cons: #

Pros:

Automatic
simple to manage
difficult to break
allows to choose when to deploy with "deploy.YES" file

Cons:

Very simple site
If you want to force a deploy you have to interact with the vm in charge of building the blog
I've not implemented any warning if something goes wrong, I can only see if it works once the site is published
I don't leverage the ability of eleventy to serve the static site locally before deploying.
the built site is not on the git repo since it will be built by a remote machine

Prerequisites: #

git
docker
rsync

Instructions: #

first of all install docker (that's your task) and pull the image we are gonna use:

docker pull femtopixel/eleventy

check that we have the image:

#docker images
REPOSITORY            TAG       IMAGE ID       CREATED       SIZE
femtopixel/eleventy   latest    6c68eb8fdb99   13 days ago   301MB

I'm keeping my eleventy work folder in sync on a git repository,
my script is in charge of checking the repo and rebuilding the site:

(every 5min via cronjob) checks if the repo has been modified, if yes:
pulls repo
checks if the file "deploy.YES: is present, if yes:
launches the container rebuilding the static site
rsyncs the static site's files to the web vm

You have to manually create the file "deploy.YES" in your elventy root folder

the script:

#!/usr/bin/env bash

#variables:
ssh_key="/root/.ssh/id_rsa"
git_repo="git@somegitserver.net:YourUser/YouRepo.git"
folder_dest="/data/blog"
blog_folder="/data/blog"
deploy_file="deploy.YES"
rsync_source="/_site/"
rsync_user="rsyncuser"
rsync_host="web_vm"
rsync_dest="/var/www/vhosts/my_blog/"
logfile="/data/blog/autoupdate_script.log"

#wake up ssh:
eval `ssh-agent`
ssh-add $ssh_key

#update the status of the repo (without pulling or modifing):
git -C $folder_dest fetch

#store the status of the repo:
git_before=$(git -C $folder_dest rev-parse HEAD)
git_after=$(git -C $folder_dest rev-parse @{u})

#compare the status before and after the fetch, if it is different it means there are change in the repo. so we will need to pull it and update the files:
if [ "$git_before" != "$git_after" ]; then
	echo "$(date +"%Y-%m-%d_%H:%M:%S") - INFO - the repo has been modified, doing my thing" | tee -a $logfile
    git -C $folder_dest reset --hard
	git -C $folder_dest pull --ff-only
	if [ -f "$folder_dest/$deploy_file" ]; then
	  echo "$(date +"%Y-%m-%d_%H:%M:%S") - INFO - deploy file is present, doing my thing" | tee -a $logfile
	  rm -rf $blog_folder/_site/*
	  echo "$folder_dest/$deploy_file exists, deploying"
	  docker run --rm -v $blog_folder:/app --name eleventy femtopixel/eleventy --output=/app/_site/
	  rsync -avhz --progress $blog_folder$rsync_source -e "ssh -i $ssh_key" $rsync_user@$rsync_host:$rsync_dest
	else
	  echo "$(date +"%Y-%m-%d_%H:%M:%S") - INFO - deploy file is NOT present, NOTE DEPLOYING" | tee -a $logfile
	  :
	fi
else
	echo "$(date +"%Y-%m-%d_%H:%M:%S") - INFO - the repo is the same as before, staying put 5 more minutes" | tee -a $logfile
	:
fi

echo "- - -" | tee -a $logfile

eval "$(ssh-agent -k)"

obviously:

chmod +x /data/autoupdate_script.sh

cron:

*/5 *	* * *	root	bash /data/autoupdate_script.sh > /dev/null

deploy.YES: #

using the file "deploy.YES" we can choose when to deploy,
the script is built so if the file is found the build and sync with the webserver take place,
otherwise the site won't be built.
This way we can continue to work on the repo and sync it, maybe working on multiple devices without publishing the site until we are ready.

Git: #

I've added the ssh key used to pull the git repo under the "Deploy keys" of the single repo,
these keys are read-only by default, so we know the only action can be a non-write one.

Web server: #

I'm using a simple apache server, it's a static site!

Conclusions: #

the site you are reading right now has been built with this flow,
so it look like it works!

Check if your public ip is soiled, also with desktop notifications

2021-04-12T00:00:00+00:00

Why: #

In the era of remote connections, Smart Working, Work From Home and Work From Anywhere, your Public IP might change often and you might end up with a "soiled" IP, an IP that has been used to spam, brute-force or other unpleasant behaviours you don't want to be associated with.

So I felt the need to check the rating of my public ip, since it might change from time to time and some cloud services might check your ip rating and raise a red flag in case you are connecting from one of those unwelcome ips.

I've written a modular script in bash so it can be implemented to generate a notification on Windows or your favourite flavour of Linux.

The script has been updated to support WSL, Linux and MacOsX out of the box,
and has been tested on WSL (v1) and Ubuntu 20.04 Desktop.

Prerequisites: #

to run the script you'll need:

curl
jq

only on Windows you'll need:

WSL (v1, not tested nor supported on WSL2)
BurntToast (see the rest of the guide for instructions)

Retrieve your ip: #

I'll use a service I love which is ifconfig.co,
it's a no-nonsense what-is-my-ip, and perfectly queryable with many tools like curl,
so if

curl http://ifconfig.co/ip

you just have your ip, ready to be used in a variable.

APIs to check ips blacklist: #

I've used abusedipdb.com:
on https://www.abuseipdb.com/ register an account and on https://www.abuseipdb.com/account/api request a new API key

For the free tier account you have 1000 checks/day (https://www.abuseipdb.com/pricing),
that translates to about to being able to do a check every 1.5minutes on a single ip,
which is way more than I need since I'll be checking every 15min for 12 hours = 48 checks/day.

So I'll be able to use the same API key with the same script on different machines,
or, way better, I can create different API keys for every script, in the end the limit of my free account is common for all my API keys.

Bash script: #

(in WSL or Linux) I've put my script in my home,
so: /home/myuser/iptest.sh

#!/bin/bash

#VARIABLES:
ip_current=$(curl ifconfig.co/ip)
#ip_current="INSERT_BAD_IP"             #this is for testing purposes, if you want to test your script with a known bad ip you can insert it here, decomment this line and comment the other ip_current. you can find a bad ip on the site https://www.abuseipdb.com/ in homepage under "Recently Reported IPs:"
ip_status="NULL"
notification="!!WARNING!!_Your_IP_is_SOILED!_"
api_key="INSERT_YOUR_API_KEY"
wsl_applogo=" -Applogo C:/Location/Of/Your/Image.png "
linux_logo=" -i /usr/share/icons/gnome/32x32/emblems/emblem-important.png "
platform='unknown'

#PLATFORM CHECK, to verify which platform the script is running on:
unamestr=$(uname -a)
if [[ "$unamestr" == *"Microsoft"* ]]; then
   platform='wsl'
elif [[ "$unamestr" == *"FreeBSD"* ]]; then
   platform='freebsd'
elif [[ "$unamestr" == *"Darwin"* ]]; then
   platform='macosx'
elif [[ "$unamestr" == *"Linux"* ]]; then
   platform='linux'
fi
#DEBUGGING (platform):
# echo $platform"


#PRECHECKS, checking if the needed software is installed:
if [[ "$platform" == "wsl" ]]; then
	echo "platform is unknown, exiting"
	exit
fi
if [[ "$platform" == "linux" ]] || [[ "$platform" == "wsl" ]] || [[ "$platform" == "macosx" ]]; then
	if ! command -v jq &> /dev/null
	then
		echo "jq not found, install jq"
		exit
	fi
	if ! command -v curl &> /dev/null
	then
		echo "curl not found, install curl"
		exit
	fi
fi
if [[ "$platform" == "wsl" ]]; then
	if ! [[ $(/mnt/c/windows/System32/WindowsPowerShell/v1.0/powershell.exe Get-Module -ListAvailable -Name BurntToast) == *"BurntToast"* ]]; then
                echo "BurntToast not found, install BurntToast"
                exit
        fi
fi


#Retrieve ip status
ip_status=$(curl -s -G https://api.abuseipdb.com/api/v2/check --data-urlencode "ipAddress=$ip_current" -d maxAgeInDays=30 -H "Key: $api_key" -H "Accept: application/json")

#Check if the output of curl is ok:
if [ "$ip_status" == "NULL" ]; then
	echo "curl failed";
	exit;
fi

#Extract the data we need from the json answer:
ip_reports=$(echo $ip_status | jq -c  '.[] | .totalReports')
ip_users=$(echo $ip_status | jq -c  '.[] | .numDistinctUsers')
ip_whitelist=$(echo $ip_status | jq -c  '.[] | .isWhitelisted')
ip_score=$(echo $ip_status | jq -c  '.[] | .abuseConfidenceScore')

#DEBUGGING (uncomment as you like):
#echo $ip_status
#echo $ip_status_clean
#echo $ip_reports
#echo $ip_users
#echo $ip_whitelist
#echo $ip_score

#Write last execution time to file (that's also debugging):
echo "updated on $(date +%Y%m%d-%H%M)" > /tmp/iptest.last

#Check the ip parameters and generate notification if over threshold:
if [ "$ip_reports" -ge "20" ] || [ "$ip_users" -ge "5" ] || [ "ip_whitelist" = "false" ] || [ "$ip_score" -ge "50" ]
then
	case $platform in
	wsl)
	   /mnt/c/windows/System32/WindowsPowerShell/v1.0//powershell.exe -command New-BurntToastNotification $wsl_applogo -Text "$notification""$ip_current"
	;;
	linux)
	  notify-send -u critical $linux_logo -t 300000 "$notification""$ip_current"
	;;
	macosx)
	  osascript -e 'display notification "It is better to change your IP" with title "YO!" subtitle "$notification""$ip_current"'
	;;
	esac
fi

NB:
you (might) need to change:

api_key="INSERT_YOUR_API_KEY" with your api key
#ip_current="INSERT_BAD_IP" (optional) with a bad ip to test notifications switching "ip_current" variables
notification variable, or anyway the text you want on the notification should not have spaces, you can workaround this issue but it wasn't one of my priority, see here: https://vepsalainen.eu/posts/5_implementing_containerized_pihole/ section "Windows 10 update notifier"
wsl_applogo variable with the location of your image on Windows
linux_logo variable has been fixed to one of the icons in the system library, which should work on different systems, fell free to change it.

variable outputs: #

these are examples of the debugging variables output in case of:

an ip that's ok:

ip_reports = 0
ip_users = 0
ip_whitelist = null
ip_score = 0

an ip that is soiled:

ip_reports = 276
ip_users = 122
ip_whitelist = false
ip_score = 100

the thresholds I set (for a period of 30days):

ip_reports >= 20
ip_users >= 5
ip_whitelist = false
ip_score >= 50

any violation of these thresholds triggers the warning.

these thresholds are a work in progress.

Windows: #

I use WSL: Windows Subsystem for Linux version 1,
since I am not using WSL2 this guide is not tested with WSL2.

Install BurntToast for notifications on windows: #

install BurntToast for notifications:

Install-Module -Name BurntToast

if needed set the execution policy as required:

#Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

then import the module and test a notification:

Import-Module BurntToast
New-BurntToastNotification -Text hello

source: https://github.com/microsoft/WSL/issues/2466

Notification image support #

if you want to add personality to your warnings you can add an image like:

New-BurntToastNotification -Applogo C:\Location\Of\Your\Image.png -Text notification_with_image

NB: the image path must be from the Windows system point of view, not WSL, hence C:...

the result will be:

auto-run Cron on WSL on Win: #

we need to enable cron so our script can run unattended,
so edit your sudoers file with

sudo visudo

and add this line:

%sudo ALL=NOPASSWD: /etc/init.d/cron start

then on Win run this command in cmd-run:

explorer.exe shell:startup

and create a new shortcut with this location:

C:\Windows\System32\wsl.exe sudo /etc/init.d/cron start

and choose a name (I choose WSL_Start_Cron)

then doubleclick your newly created shortcut to run it

if you execute

ps -ef | grep cron

now you'll see that cron is running,
I've tested rebooting and after the system comes up everything works fine.

source: https://www.linkedin.com/pulse/wsl-linux-daemons-including-cron-john-west?articleId=6730365747916365824

##configure cron to run the script:

on your shell edit your crontab:

crontab -e

add this line at the end of the file (changing "myuser" with your actual user):

*/15 8-19       * * 1-5 /home/myuser/iptest.sh

how's the schedule:
you can use the good old Crontab Guru to fiddle with it fi you are not very keen with crontab: https://crontab.guru/#/15_8-19__*_1-5

tl;dr: MON-FRI from 8 to 19 every 15min = slighlty extended work hours.

director's cut:

*/15 = every 15 minutes (on :00, :15, :30 and :45)
8-19 = all hours between 08:00 and 19:00
- = all days of the month
- = all months
1-5 = only between monday (1) and friday (5)

Notification result on Windows: #

(the ip shown in the notifications is not mine, I took one from the blacklist)

Kindle 4 NT Jailbreaking and converting to a dashboard

2021-04-07T00:00:00+00:00

Preface: #

I collect some useful metrics of my house on influxdb, and I'd like to have them ready to be glanced.

Device: #

Kindle 4th gen Non Touch (K4 NT)

Preparation: #

Donwload:

Kindle 4 JailBreak | kindle-k4-jailbreak-1.8.N-r16252.tar.xz
USBNetwork Hack | kindle-usbnetwork-0.57.N-r18392.tar.xz

from: https://www.mobileread.com/forums/showthread.php?t=225030

and unzip both files

Jailbreaking: #

follow the instructions from: https://wiki.mobileread.com/wiki/Kindle4NTHacking
which are:

Download and unzip the jailbreak.

Plug in the Kindle and copy the data.tar.gz & ENABLE_DIAGS files plus the diagnostic_logs folders to the Kindle's USB drive's root

Safely remove the USB cable and restart the Kindle (Menu -> Settings -> Menu -> Restart)

Once the device restarts into diagnostics mode, select "D) Exit, Reboot or Disable Diags" (using the 5-way keypad)

Select "R) Reboot System" and "Q) To continue" (following on-screen instructions, when it tells you to use 'FW Left' to select an option, it means left on the 5-way keypad)

Wait about 20 seconds: you should see the Jailbreak screen for a while, and the device should then restart normally

After the Kindle restarts, you should see a new book titled "You are Jailbroken", if you see this, the jailbreak has been successful.

usbNetwork: #

once you are sure everything is ok, you can install usbnetwork
connect your kindle to your pc and move the file Update_usbnetwork_0.57.N_k4_install.bin into your kindle root folder

then in kindle's settings select "Update Your Kindle"

now read carefully the file README_FIRST.txt
be careful: when the instructions say to put the commands
;debugOn and ~usbNetwork in the searchbar you can rach that by clicking the keyboard button in yor home screen (it's not options --> Search)

you can put your ssh pubkeys in /usbnet/etc/authorized_keys

and configure your /usbnet/etc/config file,
after testing everything was ok I've settled for these config:

HOST_IP=192.168.15.201
KINDLE_IP=192.168.15.244
K3_WIFI="true"
K3_WIFI_SSHD_ONLY="true"
USE_OPENSSH="false"
USE_VOLUMD="true"
QUIET_DROPBEAR="false"
TWEAK_MAC_ADDRESS="false"

where I've only modified "K3_WIFI" and "K3_WIFI_SSHD_ONLY"

the workflow I've used has been:

copy my keys in /usbnet/etc/authorized_keys
enable usbnet with the commands ;debugOn and ~usbNetwork
test that it worked ok
then enabled:
- K3_WIFI="true"
- K3_WIFI_SSHD_ONLY="true"
at this point I am able to reach the fs of the device connecting it via usb and reach it on the network via ssh
find the device's mac-address and create a reservation on the net (optional)

NB: if you don't use ssh keys it asks you for a password which should be retrievable from the serial of the device using this site: https://www.sven.de/kindle/
but unfortunately that did not work tou for me.

KAUL: #

I've tried to install KUAL but I've not succeded,
it should ease some configurations like enabling usbnet.

I didn't needed it in the end so I didn't put too much effort in it
KUAL: https://www.mobileread.com/forums/showthread.php?t=203326

Dashboard: #

I've liked this project: https://github.com/pascalw/kindle-dash
because it simplifies the process rendering a dashboard image on a remote server then retrieving that image and displaying it on the kindle.

follo the instructions here to install it: https://github.com/pascalw/kindle-dash#installation

Dashboard config: #

I've edited my configuration as follows:

export WIFI_TEST_IP=${WIFI_TEST_IP:-1.1.1.1}                                                         
export REFRESH_SCHEDULE=${REFRESH_SCHEDULE:-"*/5 7-23,0-2 * * *"}                                    
export TIMEZONE=${TIMEZONE:-"Europe/Rome"}                                                           
export FULL_DISPLAY_REFRESH_RATE=${FULL_DISPLAY_REFRESH_RATE:-4}                                     
export SLEEP_SCREEN_INTERVAL=3600                                                                    
export LOW_BATTERY_REPORTING=${LOW_BATTERY_REPORTING:-true}                                          
export LOW_BATTERY_THRESHOLD_PERCENT=10

I've edited only LOW_BATTERY_REPORTING to true, the TIMEZONE and REFRESH_SCHEDULE

Dashboard tweaks: #

update time and battery: #

see: https://github.com/pascalw/kindle-dash/issues/13

I want a line that displays the battery status and the last update time, especially in the first period of debugging.

in your /mnt/us/dashboard/local/dash.sh
find the "refresh_dashboard" function and add this line:

/usr/sbin/eips 1 39 "last update: $(date -Iminutes) battery: $(gasgauge-info -c | sed 's/%//g')"

like this:

refresh_dashboard() {
  echo "Refreshing dashboard"
  "$DIR/wait-for-wifi.sh" "$WIFI_TEST_IP"

  "$FETCH_DASHBOARD_CMD" "$DASH_PNG"

  if [ $num_refresh -eq $FULL_DISPLAY_REFRESH_RATE ]; then
    num_refresh=0

    # trigger a full refresh once in every 4 refreshes, to keep the screen clean
    echo "Full screen refresh"
    /usr/sbin/eips -f -g "$DASH_PNG"
    /usr/sbin/eips 1 39 "last update: $(date -Iminutes) battery: $(gasgauge-info -c | sed 's/%//g')"

  else
    echo "Partial screen refresh"
    /usr/sbin/eips -g "$DASH_PNG"
    /usr/sbin/eips 1 39 "last update: $(date -Iminutes) battery: $(gasgauge-info -c | sed 's/%//g')"
  fi

  num_refresh=$((num_refresh+1))
}

starting and stopping the dashboard: #

starting:

run: /mnt/us/dashboard/start.sh

stopping:

short press the power button
your device will wake up for 10sec
connect via ssh
run: /mnt/us/dashboard/stop.sh

at this point that's what you sould do to revert to normal functionality:

/mnt/us/dashboard/stop.sh
lipc-set-prop com.lab126.powerd preventScreenSaver 0
echo ondemand >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
initctl start webreader
/etc/init.d/framework start

also a device reboot works well :D

see: https://github.com/pascalw/kindle-dash/issues/12

changing the image: #

edit: /mnt/us/dashboard/local/fetch-dashboard.sh
changing the line:

$(dirname $0)/../ht -d -q -o "$1" get https://raw.githubusercontent.com/pascalw/kindle-dash/master/example/example.png

creating my own dashboard: #

dependencies:

apt-get install ImageMagic jq curl
yum install ImageMagick jq curl gnu-free-mono-fonts

I've some data on influxdb, so with API calls I can retrieve the last value of what I need:

home_temp=$(curl -s -G 'http://192.168.0.4:8086/query' --data-urlencode "db=home" --data-urlencode "q=SELECT LAST(\"value\") FROM \"temp\" WHERE \"sensor\" = 'home'")

then i can retrieve the clean value I need and round it to 1 decimal:

home_temp_cl=$(echo $home_temp | jq -r "(.results[0].series[0].columns), (.results[0].series[0].values[]) | @csv" | tail -1 | cut -d, -f2 | awk '{printf("%.1f\n", $1)}')

at this point I can create a simple txt file with all the values I need:

echo "Home temp: "$home_temp_cl > /tmp/dashboard.txt

then we can convert the txt file in an image:

convert -size 600x800 xc:white -font "FreeMono-Bold" -density 70 -pointsize 48 -gravity center -fill black -annotate +15+15 "@/tmp/dashboard.txt" /tmp/dashboard.png

NB: I've encountered some problems due to this part of the command: "@/tmp/dashboard.txt"
under 'buntu I had to edit /etc/ImageMagick-6/policy.xml
and comment out:

<policy domain="path" rights="none" pattern="@*"/>

that became:

<!--  <policy domain="path" rights="none" pattern="@*"/> -->

that was due to a security configuration of ImageMagick

under Centos the line was already disabled, and that was the default of the package.

output image:

NB: sometimes the kindle renders a white image instead of the correct one,
it seemed to be a problem of text being too near the border.
I did not understand if that was the real issue or not but fiddling with "density" and "pointsize" in convert did the trick.

Hosting the image: #

then you have to host it somewhere to make it available to the kindle
that's up to you

Final result: #

and that's the final result:

obviously my dashboard has more data,
but that's none of your beeswax.

Todo: #

[ ] modify the "dashboard" creation script to sample data for 5min then make an average (I don't want an extemporary value)
[ ] create a dashboard that is nicer, maybe with images and text, maybe using dashbling? ( https://github.com/pascalw/dashbling )
[ ] get external data like weather forecast and display it

Proxmox cluster with dishomogeneous cpu types

2021-03-15T00:00:00+00:00

TL;DR: #

NO:
for live migration within a proxmox cluster build it with same cpus

Preface: #

I have a proxmox cluster built with what I find lying around or is cheap enough to be justifiable,
so I have different cpu types in the cluster nodes: Amd Ryzen (node1) and Intel i5 (node2).

Using the default cpu type, that has a limited set of instructions, everything work as intended, particularly live migration between the hosts.

Tests: #

I've then tried to edit the custom cpu types (/etc/pve/virtual-guest/cpu-models.conf) adding a custom one that has more cpu instructions thatn the base kvm64 cpu.

But I did not find a reliable way to tell which flags were compatible with the underlying host.

I know that the best scenario is to run a cluster with homogeneous hardware, but I also know that this is what I have available :D

So, my aim is to create a custom cpu profile that contains the flags that are common to both the nodes cpus, so I can have better performance thank the standard kvm64.

So I wrote myself a script to try to match the flags that are common to both the nodes:

#clean the working file:
echo "" > result.txt

cat host1.txt | tr " " "\n" > host1_list.txt
cat host2.txt | tr " " "\n" > host2_list.txt

#loop the list of flags from node 2 against list of flags of node1 and display only the commone ones:
while read p; do
  grep -E "(^| )$p( |$)" host1_list.txt >> result.txt
done < host2_list.txt

#format the flags to be used in a proxmox custom cpus file, see: https://pve.proxmox.com/pve-docs/cpu-models.conf.5.html
# sed ':a;N;$!ba;s/\n/;+/g' : substitutes the carriage returns with ;+
# sed 's/;//' : removes the first occurrence of ; from the line
flags=$(cat result.txt | sed ':a;N;$!ba;s/\n/;+/g' | sed 's/;//')

#echo the configurations to write to: /etc/pve/virtual-guest/cpu-models.conf
echo "##############################################"
echo "cpu-model: test"
echo "    flags $flags"
echo "    phys-bits host"
echo "    hidden 0"
echo "    hv-vendor-id proxmox"
echo "    reported-model kvm64"
echo "##############################################"

#remove workfiles not needed anymore:
rm host1_list.txt host2_list.txt result.txt

Usage: #

# Proxmox_Cpu_Flags
trying to find minimum cpu flags common to 2 nodes in a cluster

## Usage:
execute lscpu on your proxmox nodes, and take the output of the line "Flags:", for example:

```
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
```
and save the content without the "Flags:" part in the files:
* host1.txt
* host2.txt

I've used host2.txt with the older cpu but it should be the same

Then execute compare.sh, you should have an output similar to:
```
##############################################
cpu-model: test
    flags +fpu;+vme;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+ht;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+constant_tsc;+rep_good;+nopl;+nonstop_tsc;+cpuid;+aperfmperf;+pni;+pclmulqdq;+monitor;+ssse3;+fma;+cx16;+sse4_1;+sse4_2;+movbe;+popcnt;+aes;+xsave;+avx;+f16c;+rdrand;+lahf_lm;+abm;+3dnowprefetch;+ssbd;+ibpb;+fsgsbase;+bmi1;+avx2;+smep;+bmi2;+rdseed;+adx;+smap;+clflushopt;+xsaveopt;+xsavec;+xgetbv1;+xsaves;+arat
    phys-bits host
    hidden 0
    hv-vendor-id proxmox
    reported-model kvm64
##############################################
```

this should be inserted on your proxmox node (one is enough) in the file: /etc/pve/virtual-guest/cpu-models.conf

## Nota bene:
right now it's not working as intended (I'm posting on r/proxmox to ask for help)

But it's not working as hoped:

I've tried to boot the VM on Node1 using the new cpu profile gives me many errors on cpu flags incompatible:
I had to remove:aperfmperfconstant_tsccpuidnonstop_tscnoplrep_goodhtmonitor
once I removed the incompatible cpu flags on AMD it works ok
I've tried to migrate to Node2 the vm and it crashed due to non compatible cpu flag:
I had to remove:ssbd

so I tried again to start the vm on Node1 and migrate to Node2, this time I got a nice kernel panic, which is the same output I get when I just start the vm on Node2 without the migration.

image of kernel panic:

So there's obviously something wrong in my process, but I am not understanding what, maybe the compatibility is not supported among different cpu types or I am starting from some very wrong assumptions.

I've found another thread where OP selected Ivy-Bridge or Sandy-Bridge to enhance compatibility for migration between the hosts he/she had (Amd and Intel),my question is:is there a reliable way to determine the highest cpu flags/cpu profile supported by both machines?(other than trial and error selecting the various cpu types and trying a live migration)

Reddit post and suggestions: #

I've posted a questione on r/proxmox: post
and u/elettronik gave me some insights I flew over.

Live migration for me is not essential, since most of the projects I host are not that sensitive to a soft and controlled shutdown and re-power on, so it definitely could be a way.

After another bit of fiddling and trying I've managed to fix what I think are the address size in the config file, setting it to the lowest value:

cpu-model: test3
flags +fpu;+de;+pse;+tsc;+msr;+pae;+mce;+cx8;+apic;+sep;+mtrr;+pge;+mca;+cmov;+pat;+pse36;+clflush;+mmx;+fxsr;+sse;+sse2;+syscall;+nx;+pdpe1gb;+rdtscp;+lm;+pni;+pclmulqdq;+ssse3;+cx16;+sse4_1;+sse4_2;+movbe;+popcnt;+aes;+xsave;+rdrand;+lahf_lm;+abm;+fsgsbase;+bmi1;+smep;+bmi2;+adx;+smap;+clflushopt;+xsaveopt;+xgetbv1;+arat;+hypervisor
phys-bits 39
hidden 0
hv-vendor-id proxmox
reported-model kvm64

The flags are tested with:

qemu-system-x86_64 -cpu qemu64,check,+fpu,+de,+pse,+tsc,+msr,+pae,+mce,+cx8,+apic,+sep,+mtrr,+pge,+mca,+cmov,+pat,+pse36,+clflush,+mmx,+fxsr,+sse,+sse2,+syscall,+nx,+pdpe1gb,+rdtscp,+lm,+pni,+pclmulqdq,+ssse3,+cx16,+sse4_1,+sse4_2,+movbe,+popcnt,+aes,+xsave,+rdrand,+lahf_lm,+abm,+fsgsbase,+bmi1,+smep,+bmi2,+adx,+smap,+clflushopt,+xsaveopt,+xgetbv1,+arat,+hypervisor

which with the flag "check" outputs errors in case the flags are not supported on the host, so this seems a reliable way to determine which flags are supported and which not.

So:
dual cpu works ok also on Ryzen, live migration is ok to Intel but the other way around the kernel panics.

Conclusions: #

Right now I've decided to stick with the default kvm64 for all the vms until I'll need some more performance or on the specific vms that will benefit from a broader set of cpu instructions, in the near future I will evaluate the possibility to apply a broader set of instructions but losing live migrations.

So, for reliable production hardware stick with Proxmox's manual and create cluster with same cpu types on the hosts,
otherwise you know what your problems might be.

Bonus: #

benchmark of different cpu flags:

128/256 F = with AES flag
128/256 N = without AWS flag

	128 F	256 F	128 N	256 N	128 FvsN	256 FvsN
16B	662910	609306	154295	116026	4.2x	5.2x
64B	1414566	1040415	173676	127442	8x	6x
256B	1460026	1069234	187151	131995	7.8x	8x
1KB	1476740	1075118	375774	272275	4x	4x
8KB	1475370	1073244	383018	278899	3.8x	3.8x
16KB	1478732	1073479	383800	280018	3.8x	3.8x

test run as

openssl speed -evp aes-128-cbc
openssl speed -evp aes-256-cbc

obviously it's a single test but one can see the improvement.

Surface Go Dual Boot Secureboot

2020-11-11T00:00:00+00:00

Preface: #

The goal of this guide is to have Windows 10 and Ubuntu installed on a Surface device (I've tested this on the Go and should work the same on the Go2),
with secureboot enabled for both of them.

The version used in this guide are:

Win10 20H2 Pro (Pro has full bitlocker support)
Ubuntu Mate 20.04.1 LTS

Prerequisites: #

disabling Fast Startup:
- in: Control Panel --> hardware and Sound --> Power Options --> System Settings
  select "Choose what the power buttons do"
  and then unlock the buttons on the bottom clicking on "Change Settings that are not curently available"
  and deselect "Turn on gast start-up (recommended)"
  then click on "save changes"
surface Device installed with Win10
usb pen with Ubuntu 20.04 on it:
- you should create the drive downloading the ISO and dd'ing it on the drive:
  dd if=ubuntu.iso of=/dev/sdX bs=1M status=progress,
  in this way you are shure to have a UEFI boot drive
  I've used Ubuntu Mate 20.04.1 LTS
patience

Win10 pre-tasks: #

If enabled, disable bitlocker from Win10 bitlocker menu,
wait for the decryption to finish.

Shrink the windows partition form the disk manager to leave space for ubuntu.

Reboot in advanced mode to boot from the USB drive with ubuntu:
hold "shift" while clicking on "reboot",
you'll be in windows recovery mode, select "other disk" and select "Linpus ..." to boot in ubuntu

Ubuntu Live tasks: #

Shrinking: #

fire up utubntu live and with gparted move Win10's recovery partition back to leave all the empty space in the end:

before:

after:

Partitioning: #

with gparted create the new partitions as follows:

5: efi partition in fat32, flags boot and esp (will be mounted on /boot/efi)
6: /boot partition in ext2 (will be mounted on /boot)
7: luks partition

Number  Start   End     Size    File system  Name                          Flags
 1      1049kB  274MB   273MB   fat32        EFI system partition          boot, esp
 2      274MB   408MB   134MB                Microsoft reserved partition  msftres
 3      408MB   95,1GB  94,7GB  ntfs         Basic data partition          msftdata
 4      95,1GB  96,1GB  1074MB  ntfs         Basic data partition          hidden, diag
 5      96,1GB  96,7GB  537MB   fat32        EFI System Partition          boot, esp
 6      96,7GB  97,2GB  537MB   ext2
 7      97,2GB  128GB   30,8GB

Luks: #

format the 7th partition that we'll use with luks:

cryptsetup luksFormat /dev/nvme0n1p7
cryptsetup luksOpen /dev/nvme0n1p7 CRY_main

pvcreate /dev/mapper/CRY_main
vgcreate VG_main /dev/mapper/CRY_main

lvcreate -L 15G -n LV_root VG_main
lvcreate -L 2G -C y -n LV_swap VG_main

vgchange -a y

format the EFi partition:

mkfs.vfat /dev/nvme0n1p5

Ubuntu installer: #

now fire up the ubuntu common installer from the desktop icon,
and continue with the normal selection:

minimal installation
download updates while installing ubuntu mate
install third party software...
- configure secureboot
  - choosing a password of your choosing

the last part is very important since it will allow you to enroll the newly created MOK key in the machine's UEFI

in the partitioning part do:

select efi nvme0n1p5 and select "Use as" "EFI System Partition"
select boot nvme0n1p6 and select "Use as" "ext2" and mount on /boot, do format
select /dev/mapper/VG_main-LV_root "Use as" "ext4" and mount on /
select /dev/mapper/VG_main-LV_swap "Use as" swap
select install the bootloader on the efi partition: nvme0n1p5

and continue the installation

DO NOT REBOOT
DO NOT REBOOT
DO NOT REBOOT

Post-installation tasks: #

once finished do not reboot,

mount the newly created system on /mnt creating the environment for chrooting correctly:

mount /dev/mapper/VG_main-LV_root /mnt
mount /dev/nvme0n1p6 /mnt/boot
mount /dev/nvme0n1p5 /mnt/boot/efi

mount --bind /dev /mnt/dev
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys

then chroot in to the newly prepared folder:

chroot /mnt

In the chroot: #

blkid | grep -i luks

take the field "UUID"
ES:

/dev/nvme0n1p7: UUID="11111111-2222-3333-4444-555566667777" TYPE="crypto_LUKS" PARTUUID="aaaaaaaa-bbbb-cccc-dddd-eeeeffffaaaa"

in this case it's: 11111111-2222-3333-4444-555566667777

edit the file /etc/crypttab (if not present create)
and add:

CRY_main UUID=11111111-2222-3333-4444-555566667777 none luks,discard

NOTE: the name shall be the same as you opened the crypt device in steps

then recrate the system boot environment:

grub-install
update-initramfs -u -k all

and exit the chroot with a simple

exit

now you can reboot your system.

be very aware of the next steps!

Enrolling MOK Key to Secureboot: #

you will be greeted by the UEFI Key manager, where you will be able to enroll the new key for secureboot:

choose "Enroll MOK"

then "continue"

select "Yes"

then insert the password you selected during ubuntu installation

and then select "reboot"

After the reboot: #

now the system should restart and you should be greeted by grub's selection screen,
there you can choose to boot in Ubuntu or in Windows (which has been detected by the installer)

Re-activating bitlocker #

after booting to windows you can re-activate bitlocker,
maybe setting a custom passphrase at boot so your system won't load automatically without your input.
see: Win10 - Surface Go - Bitlocker with TPM and advanced pin

Debian 9-13 unlock luks root at boot via ssh

2020-09-14T00:00:00+00:00

based on: https://stinkyparkia.wordpress.com/2014/10/14/remote-unlocking-luks-encrypted-lvm-using-dropbear-ssh-in-ubuntu-server-14-04-1-with-static-ipst/

updated for debian 12 (2024-12-04) and tested on debian 13 (2025-12-02) #

NB: in an upgrade from 11 to 12 if you tell the installer not to overwrite your configfiles averything will be moved to the new file names and work correctly out-of-the-box

apt install -yy dropbear-initramfs cryptsetup-initramfs lvm2

add your keys to: /etc/dropbear/initramfs/authorized_keys

nano /etc/dropbear/initramfs/authorized_keys

add your network interface config to: /etc/initramfs-tools/initramfs.conf

nano /etc/initramfs-tools/initramfs.conf

DEVICE=eth0
IP=192.168.0.10::192.168.0.1:255.255.255.0:your-hostname:eth0:off

to automatically call the script to unlock your disk upon entering dropbear add the line to this file:
(source: https://www.arminpech.de/2019/12/23/debian-unlock-luks-root-partition-remotely-by-ssh-using-dropbear/)

nano /etc/dropbear/initramfs/dropbear.conf

DROPBEAR_OPTIONS="-RFEsjk -c /bin/cryptroot-unlock"

and finally update initramfs:

update-initramfs -k all -u

old versions (up to debian 11) #

apt-get install -yy dropbear-initramfs cryptsetup-initramfs lvm2

add your keys to: /etc/dropbear-initramfs/authorized_keys

nano /etc/dropbear-initramfs/authorized_keys

add your network interface config to: /etc/initramfs-tools/initramfs.conf

nano /etc/initramfs-tools/initramfs.conf

DEVICE=eth0
IP=192.168.0.10::192.168.0.1:255.255.255.0:your-hostname:eth0:off

nano /etc/dropbear-initramfs/config

DROPBEAR_OPTIONS="-RFEsjk -c /bin/cryptroot-unlock"

and finally update initramfs:

update-initramfs -k all -u

END #

NOTES for particular cases: #

device drivers: #

sometimes, in case you are using exotic net drivers, you'll need to add your device modules to initram modules:

echo $(while read m _; do \
/sbin/modinfo -F filename "$m"; done </proc/modules |sed -nr \
"s@^/lib/modules/`uname -r`/kernel/drivers/net(/.*)?/([^/]+)\.ko\$@\2@p")   >> /etc/initramfs-tools/modules

then edit /etc/initramfs-tools/modules since the modules will be on the same line:

tap r8169 realtek

tap
r8169
realtek

after upgrading, dropbear's net won't work anymore: #

I had this problem on a specific system: a Proxmox based debian 10 updated to 11 (and proxmox 6.4 to 7.2).
After the update I could not unlock the machine remotely, after a lot of debugging I've settled on this "fix":
changing the ip address....

I've always used the same IP address for unlocking via dropbear and for the system,
but for some obscure reason this isn't working anymore.

so the system for example will have 192.168.0.10 and the dropbear unlock ip: 192.168.0.11

To add obscurity to this issue, I have another machine that is working perfectly with the same ip on os & dropbear.

This behaviour is confirmed also after change subnet to the server.

I have no idea why.

OLD GUIDE:
until mid 2020 I've used this script to unlock the disk:

to automatically unlock the disk with the command "unlock" add this file:

nano /etc/initramfs-tools/hooks/crypt_unlock.sh

#!/bin/sh
#
# By Stinky Parkia
# https://stinkyparkia.wordpress.com/2014/10/14/remote-unlocking-luks-encrypted-lvm-using-dropbear-ssh-in-ubuntu-server-14-04-1-with-static-ipst/

PREREQ="dropbear"

prereqs() {
    echo "$PREREQ"
}

case "$1" in
    prereqs)
    prereqs
    exit 0
    ;;
esac

. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions

if [ "${DROPBEAR}" != "n" ] && [ -r "/etc/crypttab" ] ; then
    cat > "${DESTDIR}/bin/unlock" << EOF
#!/bin/sh
if PATH=/lib/unlock:/bin:/sbin /scripts/local-top/cryptroot; then
    kill \`ps | grep cryptroot | grep -v "grep" | awk '{print \$1}'\`
    # following line kill the remote shell right after the passphrase has
    # been entered.
    kill -9 \`ps | grep "\-sh" | grep -v "grep" | awk '{print \$1}'\`
    exit 0
fi
exit 1
EOF

    chmod 755 "${DESTDIR}/bin/unlock"

    mkdir -p "${DESTDIR}/lib/unlock"
    cat > "${DESTDIR}/lib/unlock/plymouth" << EOF
#!/bin/sh
[ "\$1" == "--ping" ] && exit 1
/bin/plymouth "\$@"
EOF

    chmod 755 "${DESTDIR}/lib/unlock/plymouth"

    echo To unlock root-partition run "unlock" >> ${DESTDIR}/etc/motd
fi

Debian 10 Buster various problems

2020-09-14T00:00:00+00:00

Preface #

I've always used Debian
And I've loved it
because it is was straightforward and you could easily solve your issues searching the net.

Issue #

I had some issues because some common (from 20years to now) commands were no longer available,
commands like

reboot
shutdown
iptables-whatever
etc..

and it seemed pretty strange.

Workaround #

I found out that all these executables are already present in the folder /sbin or /usr/sbin,
but these folder are not in the PATH variable.

searching a bit it should be due to /etc/profile:

# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "`id -u`" -eq 0 ]; then
  PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
else
  PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"
fi
export PATH

if [ "${PS1-}" ]; then
  if [ "${BASH-}" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "`id -u`" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi

which will load the second line of PATH and not the first one that includes our much needed /sbin and /usr/sbin

the solution is to add to
/etc/environment

PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"

which will fix the system, at least for my needs.

Ansible task #

Ansible task to achieve that:

    - name: Fix Debian10's shitty executables paths
      lineinfile:
        dest: /etc/environment
        line: 'PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"'
        state: present
      when: ansible_distribution == 'Debian' and ansible_distribution_major_version == '10'

so, the issue is fixed, but please Debian, ripijate.

Update #

As a wise friend of mine pointed out, changing user with

su root

is not the same as doing

su - root

since it won't open a new shell and use the correct variables.
more here:
https://unix.stackexchange.com/questions/15611/what-is-the-difference-between-su-and-su-root

so 50/50 contributory negligence, ball in the center.

Dumb-migrating from a static Wordpress to Eleventy

2020-08-30T00:00:00+00:00

My first blog was a wordpress installation on a hosted site,
then it got moved to a full vm under my control,
then it got shutdown and forgot for some months,
then it was recovered and lived for brief moments just when writing entries because the "live" site was actually a static export on S3.

Then today, wordpress has been decommissioned 'cause it was useless to keep something alive when the hassle of using it was more than the usefullness of it.

So I've migrated the old wordpress to Eleventy using eleventy-base-blog,
this allows me to edit the blog entries easily from CLI, preview it locally and upload the site with a simple "aws s3 sync".

The migration has been done without automatic tools,
all the entries have been dumb-migrated by hand, and it shows since the formatting has to be fixed on many pages.

Installation: #


sudo apt-get install nodejs
sudo apt-get install npm
npm init -y
npm install @11ty/eleventy --save-dev

Running and serving: #

running eleventy to build the static site:

npx @11ty/eleventy

to run and serve eleventy locally:

npx @11ty/eleventy --serve

and you'll find your site on localhost:8080

TODO:

fix formatting

Win10 - Surface Go - Bitlocker with TPM and advanced pin

2019-03-25T00:00:00+00:00

Preface: #

The aim of this article is to configure Bitlocker to ask for a password on boot so the protected system will not be avilable without user interaction.

This is updated to 2020-11-11, on Win10 20H2

NB: I am not advocating the use of bitlocker instead of other approaches, nor I am saying that it's secure.
I am writing this guide for cases where you are forced or need to use a device with Win10, and are able to at least make it a bit more secure or more difficult for an attacker to access your data.

Modify group policies: #

execute gpedit.msc to edit the Bitlocker policies and go to:

Computer Configuration
Administrative Templates
Windows Components
BitLocker Drive Encryption
Operating System Drives

then edit:

Require additional authentication ad startup
Enable use of bitlocker authentication requiring preboot keyboard inputs
Allow enhanced PINs for startup
- this allow us to use alphanumeric long "passphrases" instead of short numeric only PINs

I've found out that the "Enable use of bitlocker authentication requiring preboot keyboard inputs" is still needed also if the newer firmwares allos the tablet to show a keyboard when selecting the bitlocker input field with the touchscreen.

Encrypting the drive: #

Then open the bitlocker utility from the control panel and click on "Turn on bitlocker":

Then you will be able to choose how you want to unlock your boot drive,
choose PIN and enter your pin (sorry, no screenshot)

The choose how you want to save your recovery key, needed to unlock the drive in case you loose your pin or the encryption has some problems.

Dualbooting: these problems might arise in case ubuntu isn't installed correctly with secureboot, as I did one time,
since when Ubuntu upgrades the kernel and it's boot sequence, Windows detect this as something broken in the secureboot process and asks you for the recovery key.
This article should solve this issue: Surface Go Dual Boot Secureboot

then choose to encrypt the entire drive (the screenshot has the wrong selection):

and choose the newer encryption method:

you will be asked if you want to follow a pre-encryption check to be sure that the newly encrypted drive will work as intended, I choose yes (sorry, no screenshot)

Your system will reboot,
you will be asked to unlock the drive:

and the system will boot.
At this point the drive will encrypt, you can monitor the status of the process from the tray:

then you are set.

FASTBOOT “TOO OLD” AND “ERROR CANNOT GENERATE IMAGE FOR USERDATA”

2019-03-24T00:00:00+00:00

When you try to install a factory image on a newer Android device (in my case a Google Pixel “1”), you might encounter this error which i sude to the android platform tools which are too old.

Resolution:
download the newer platform-tools from https://developer.android.com/studio/releases/platform-tools

Decompress them with:

unzip platform-tools_r28.0.2-linux.zip

a new platform-tools folder will be created
then in your folder where you have unzipped the factory default edit flash-all.sh and add this line on top:

export PATH=/home/YOURUSER/Downloads/platform-tools:$PATH

in this way the script will search for executables in that folder, using the newer ones.

NEXUS 5X LINEAGEOS 15.1 – “VENDOR IMAGE MISMATCH HAS BEEN DETECTED”

2018-07-15T00:00:00+00:00

I’ve updated LineageOS on a colleague’s Nexus5x and I followed these steps:

Downloaded the latest (8.1.0 (OPM6.171019.030.E1, Jul 2018)) stock image for nexus5x from Google repos
updated the 5x using the flash-all.sh included in the archive
following lineageos tutorial for N5x I’ve
⋅⋅* downloaded the latest (lineage-15.1-20180702-nightly-bullhead-signed.zip) image from the repo
⋅⋅* downloaded the latest Open Gapps pico package for ARM64, android 8.1
⋅⋅* downloaded the and installed twrp
⋅⋅* flashed my device with lineageos
⋅⋅* flashed opengapps

when I restarted I was greeted with a message stating

“A vendor image mismatch has been detected. Typically this means your vendor image is out of date. Please ensure your vendor image matches OPM6.171019.030.B1”

NB: the following image states “OPM6.171019.030.E1” wince is the latest warning I received.

this was because the LineageOS image I downloaded was the last one available but was built on OPM6.171019.030.B1 which wasn’t the latest google image available.

you can find this info in the changelogs:

starting from lineage-15.1-20180618-nightly-bullhead-signed.zip it was built upon OPM6.171019.030.B1(full changelog, detailed changelog on our issue)
starting from lineage-15.1-20180709-nightly-bullhead-signed.zip it was built upon OPM6.171019.030.E1 (full changelog, detailed changelog on our issue)

Solution:
flash the vendor image:

from the google repos download the package corresponding to the vendor image your N5x wants
extract the archive to a folder, move into that folder then unzip the zipfile relative to your version, mine: “image-bullhead-opm6.171019.030.b1.zip”
you’ll find a file named “vendor.img”

put the N5x in fastboot and flash said file with:

fastboot flash vendor vendor.img

reboot your N5x

fastboot reboot

the issue should be solved and you should receive no error when booting.

I’ve encountered this problem twice, since I installed LineageOS lineage-15.1-20180702 and then updated to lineage-15.1-20180709 and the two images have been built on different vendor image versions.

FakeKitten, or how to trick Amazon Prime Photos to store anything

2018-01-05T00:00:00+00:00

Scope: #

Some have an Amazon Prime with Unlimited Photo Storage,
it would be a pity to use it just for photos.

How: #

The script creates a new file and adds the photo of a very cute kitten at the end of it.

In this way the file gets recognized as a photo and won't count against your usable space (this is after uploading 3.3GB of "fakekitten-ed" ubuntu iso):

In the encoding phase useful data gets stored in the destination filename,
this data will be used for decoding and checking that the image is consistent and not corrupted.

Unfortunately the upload will be done manually since the APIs for Amazon Drive are not available anymore [0] and we won't be able to use automated tools.
More info on the incident that allegedly kicked off the API removal [1]

Encoding: #

stores imagesize of the kitten image
creates sha1 checksum of the original file
creates filename with useful data: "FakeKitten_JpgImgSize_OrigFileName_OrigFileSha_ImageFilename"
- FakeKitten: fixed text
- JpgImgSize: the filesize of the kitten.jpg image
- OrigFileName: filename of the original file
- OrigFileSha: sha1sum of the original file
- ImageFilename: image filename (kitten.jpg)
appends the image at the end of the file

Decoding: #

retrieves the variables from the filename
creates new file skipping the appended image at the end of the file
creates sha1 checksum of the downloaded file and checks it against the filename

Prerequisites: #

You will need:

something that runs bash
kitten image
the files you want to upload
obv. an Amazon Prime account to upload the "photos"

Instructions: #

just run the script with the options:

Encode: ./AmazonPrimeWhatever.sh fileName encode
Decode: ./AmazonPrimeWhatever.sh fileName decode

The script: #

#!/usr/bin/env bash

if [ -z "$1" ] || [ -z "$2" ] || [[ $2 != "encode" && $2 != "decode" ]]
	then echo "launch the script with the desired filename and operation"
	echo "./AmazonPrimeWhatever.sh FILENAME OPERATION(encode or decode)"
	exit
fi

image="kitten.jpg"
imagesize=$(du -b $image | cut -f1)
origsha=$(sha1sum $1 | cut -d " " -f1)
destimage="FakeKitten_"$imagesize"_"$1"_"$origsha"_"$image


if [ $2 = "encode" ]
	then cp $image $destimage
	dd if=$1 bs=1M >> "$destimage"
	echo "encode completed in $destimage"
fi

if [ $2 = "decode" ]; then
	imagebs=$(echo $1 | cut -d "_" -f2)
	origname=$(echo $1 | cut -d "_" -f3)
	origsha=$(echo $1 | cut -d "_" -f4)
	origimage=$(echo $1 | cut -d "_" -f5)
	dd if=$1 bs=1M skip=$imagebs iflag=skip_bytes > "$origname"
	echo "decode completed in $origimage, checking file integrity"
	echo $origsha"  "$origname > $origname".sha1"
	shaoutput=$(sha1sum -c $origname".sha1")
	echo $shaoutput
	if [[ $shaoutput != *"OK"* ]]; then
		echo ""
		echo "!!! FAILED SHA VERIFICATION!!! EXITING"
		echo "!!! DELETING ALL CREATED FILES !!!"
		rm $origname
		rm $origname".sha1"
		exit
	fi
	rm $origname".sha1"
fi

echo ""
echo "end of my job"

MAKE VMWARE-VIEW work ON LUBUNTU 16.04 32BIT

2017-02-02T00:00:00+00:00

vMware View scan output:

VMware Horizon Smart Card
        Success
VMware Horizon Real-Time Audio-Video
        Success
VMware Horizon Client Drive Redirection
        Success
VMware Horizon Multimedia Redirection (MMR)
        Failed          libgstapp-0.10.so.0
        Failed          libgstbase-0.10.so.0
        Failed          libgstreamer-0.10.so.0
VMware Horizon PCoIP
        Success
VMware Horizon USB Redirection
        Success
VMware Horizon Virtual Printing
        Success
VMware Horizon Client
        Success

some of the steps:

apt-get install libglibmm-2.4-1v5 libglibmm-2.4-dev libffi-dev libffi6

ln -s /usr/lib/i386-linux-gnu/libffi.so.6 /usr/lib/i386-linux-gnu/libffi.so.5

some of the guides followed:
https://nchrissos.wordpress.com/2015/05/26/the-odyssey-of-building-a-lightweight-vmware-view-linux-workstation-for-horizon-6/
https://communities.vmware.com/thread/499473
http://askubuntu.com/questions/518644/error-while-loading-shared-libraries-libffi-so-5

Debian 7.11 on Powermac G5

2017-01-15T00:00:00+00:00

Downloaded Debian 7.11 netinstall from here, burnt on a DVD and inserted in the superdrive.

To open the superdrive witha non-mac keyboard, press F12 at boot.
To boot from the cdrom press “c” at boot.
You’ll be greated by the open firmware, select “c” to boot from cdrom:

Then type “install” and press enter to continue with the installation

Then proceed with a normal installation.

Note that the installer has a red background for the whole duration of the installation,
I’ve seen this on x86 when there was an error but that doesn’t seem the case since the installation proceeded and completed fine.

Normal selecting the right nic:

Normal partitioning the disks:

And almost normal selecting a mirror:
when selecting the network mirror for the installation I had to select Germany –> ftp.de.debian.org because the Italian one (ftp.it.debian.org) hadn’t the powerpc architecture available.

However selecting the german mirror I received this error (that I received also using the UK mirror) and continued:

Selected the software:

Then finished the installation without further problems.

The system (with a single disk with Debian) boots automatically.

I incurred in a graphical problem where I couldn’t see about half of the screen, either in graphical or console mode:

The issue was due to the fact that I was using a DVI-VGA adapter, once I connected the PowerMac via DVI to the monitor the problem solved:

root@DebPowMacG5:~# lscpu
Architecture: ppc64
Byte Order: Big Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
NUMA node(s): 1
Model: PowerMac11,2
L1d cache: 32K
L1i cache: 64K
L2 cache: 1024K
NUMA node0 CPU(s): 0,1

Useful read: http://larryhynes.net/2014/11/installing-debian-on-powermac-g5.html

KERNEL 586 NO HT

2017-01-13T00:00:00+00:00

doing a fresh install with cbpp-1.0-i386-20150428.iso on Atom N270 (single core, HT), had a problem due to the HT that was not being used.

This was due to the fact that kernel 3.16.0-4-586 does not have SMT built in kernel:

user@host:~$ cat /boot/config-3.16.0-4-586  | grep CONFIG_SCHED_SMT

receiving no output,
I updated to kernel 3.16.0-4-686-pae and then the HT support was enabled:

user@host:~$ cat /boot/config-3.16.0-4-686-pae  | grep CONFIG_SCHED_SMT
CONFIG_SCHED_SMT=y

install the kernel with:

apt-get install linux-image-3.16.0-4-686-pae

Linux user SCP only

2017-01-08T00:00:00+00:00

install rssh:

apt-get install rssh

then to add a user:

useradd -m -d /home/didi -s /usr/bin/rssh didi

edit

nano /etc/rssh.conf

and select how much users are locked out:

allowscp
#allowsftp
#allowcvs
#allowrdist
#allowrsync
#allowsvnserve

source: https://www.cyberciti.biz/tips/linux-unix-restrict-shell-access-with-rssh.html

Soluzione [quasi] completamente libera (ed economica) per lo streaming/registrazione live

2016-12-01T00:00:00+00:00

Il Problema: #

Dopo 4 anni di riprese ai LinuxDay di ERLUG dove a volte il risultato migliore che si riusciva ad ottenere a livello video era questo:

ho iniziato a cercare una soluzione per acquisire l’output del monitor del relatore in modo che fosse utilizzabile per una successiva consultazione dei video registrati.

Requisiti: #

poter gestire uno o piu’ input video
essere remotizzabili il piu’ possibile
essere supportati da sistemi operativi opensource
essere economici
essere una soluzione agnostica non invasiva per il relatore permettendoci di prendere il segnale che sarebbe poi andato al proiettore

La mia ricerca mi ha portato alle schede di acquisizione USB o Pci ma spesso violavano tutti i punti.

La Soluzione: #

L’epifania e’ arrivata leggendo questo articolo dove Danman, l’autore, ha preso un extender HDMI che trasporta il segnale via ethernet, ne ha reversato il protocollo e ha creato uno script per poter ricevere lo stream audiovideo con VLC.

Dato che la coppia di ripetitori (TX e RX) e’ costa circa 70euri su Amazon, il punto 4 e’ perfettamente rispettato.

I primi test con l’extender si e’ rivelato un fallimento, vedevo lo stream con tcpdump ma non ricevevo niente, il problema era legato al fatto che gli adattatori su cui Danman ha fatto lo script erano v1, i miei v3.
Dopo qualche ricerca capito sull’aggiornamento del blog di Danman, e magia e stupore, gli adattatori sono stati aggiornati al protocollo HDbitT, in sintesi lo stream UDP creato dal trasmettitore e’ nativamente interpretabile da vlc:

vlc udp://@239.255.42.42:5004

La scritta “Please check the TX input signal” e’ normale dato che non avevo ancora attaccato una sorgente HDMI al tramsettitore, vuol dire che funziona!

La qualita’ e’ buona, un fattore da considerare e’ che il segnale audiovideo arriva con un ritardo di 1,5secondi circa, va compensato in caso si usino altre fonti audio.

Uno dei vantaggi del fatto che gli adattatori lavorino su ethernet con uno stream multicast udp e’ quello che il cavo di collegamento tra l’adattatore e il pc che ne prende lo stream e’ fino a 120mt.
Oppure e’ possibile attaccarlo ad uno switch o ad un bridge wireless.

Gli adattatori: #

Gli adattatori che ho provato io sono gli AGPteck LKV373 e gli ESYNiC.

I secondi sono stati presi alla cieca in quanto di forma nettamente diversa dai primi, che sono stati reversati da Danman,
la cosa da guardare e’ che nella descrizione del prodotto la massima estensione di segnale su cavo CAT5/CAT6 sia di 120/125mt e NON di 60mt, poiche’ nel primo caso si tratta di adattatori ethernet, nel secondo di adattatori che usano il cavo CAT5/CAT6 solo come media di trasmissione ma NON usano il protocollo ethernet.

in fondo al post ci sono le foto dell’adattatore ESYNiC.

Come bonus dovrebbe essere possibile usare il ricevitore semplicemente mandandogli uno stream udp come farebbe il trasmettitore.

Entrambi i modelli si alimentano a 5V in continua.

L’evoluzione: #

Lo step successivo e’ stato quello di trovare un software che permettesse di fare regia in tempo reale, streaming e registrazione su disco.
OBS Studio ci ha dato quasi tutto questo, le limitazioni riscontrate con OBS Studio sono:

nessun monitor del segnale audio sull macchina su cui gira OBS. e’ un qualche bug di cui non sono riuscito a fare il workaround
un solo streaming alla volta
a volte e’ solido come una roccia altre volte crasha perche’ lo hai guardato storto
folta community su windows e mac che lo usa per streamare le videogiocate e miriadi di plugin per windows, pochi per linux (ma c’e’ da dire che hanno unificato OBS per tutte le piattaforme Win/Mac/Lin di recente quindi mi aspetto che la situazione evolva)
non legge nativamente gli stream udp, come quello che sono andato ad utilizzare (pare che alcuni siano riusciti a fargli leggere lo stream, la soluzione che ho trovato e’ stata aprire lo stream con VLC e fatto acquisire ad OBS la finestra di xserver)
il verdetto e’ che OBS e’ stato un ottimo software per fare streaming e registrazione contemporanea, l’interfaccia e’ molto intuitiva:

La topologia dei collegamento che e’ stata utilizzato per il NinuxDay2016 (http://wiki.ninux.org/NinuxDay2016):

Conclusioni: #

Questa soluzione a livello di qualita’ audiovideo e’ ampiamente soddisfacente, il delay del video rispetto all’audio e’ facilmente gestibile dal mixer di OBS.

Il “[quasi] completamente open” del titolo del post e’ relativo al fatto che il firmware degli extender e’ con buone possibilita’ non open ma riflettendo ilo stesso problema si sarebbe proposto ugualmente con delle schede di acquisizione.

Una cosa da calcolare bene e’ la banda in upload, meta’ dello streaming e’ stata fatta a 2400kbps (+160 di audio), un’altra meta’ a 1200 (+160), va calcolato che serve una linea buona, non per forza dedicata ma il meno congestionata possibile.

TODO: #

verificare la reale necessita’ di potenza di calcolo della GPU
risolvere il problema del monitor dell’audio su OBS
far leggere ad OBS direttamente lo streaming udp
imparare e farsi un server di streaming autonomamente in modo da non appoggiarsi a servizi commerciali
testare il ricevitore con uno stream udp da pc

Videi: #

I video registrati al NinuxDay2016 con questo sistema sono visibli sul canale youtube di Ninux.

Foto adattatore ESYNiC

Loop mounting an encrypted luks full disk image

2016-11-02T00:00:00+00:00

Open:

fdisk -l -u imagename.img

kpartx -a -v imagename.img

cryptsetup luksOpen /dev/mapper/loop0pN LUKSNAME

vgscan

vgchange -a y VGNAME

mount /dev/mapper/VGNAME-LVNAME MOUNTPOINT

Close:

umount MOUNTPOINT

vgchange -a n VGNAME

cryptsetup luksClose LUKSNAME

dmsetup info
dmsetup remove /dev/mapper/loop0p1
dmsetup remove /dev/mapper/loop0pN…

losetup -a
losetup -d /dev/loop0

source: http://www.blaicher.com/2013/01/accessing-an-encrypted-full-disc-image-lukslvm/

UBUNTU 16.04 SINGLE INSTALL ON MACBOOKAIR 2014

2016-11-01T00:00:00+00:00

NB: these are notes for my particular installation, YMMV

Installation:
“burn” the image on the usb stick, I tend to use usb-creator-gtk on ubuntu, some say dd-ing the image on the usb is ok,
insert the usb key on your applething, power on and press opt/alt: screenshot-from-2016-11-02-22-49-46
then select “EFI boot” and go on with the installation

Keyboard:
this is the holy grail: https://github.com/free5lot/hid-apple-patched
it permits you to finally swap the “fn” and “ctrl” keys.

install:

git clone https://github.com/free5lot/hid-apple-patched
cd hid-apple-patched
sudo dkms add .
sudo dkms build hid-apple/1.0
sudo dkms install hid-apple/1.0

edit the config file:

sudo nano /etc/modprobe.d/hid_apple.conf

add:

options hid_apple fnmode=2
options hid_apple swap_fn_leftctrl=1
options hid_apple swap_opt_cmd=1

apply and reboot:

sudo update-initramfs -u
sudo reboot

alternatively you can follow this guide, without the swapping of “fn” and “ctrl”: https://help.ubuntu.com/community/AppleKeyboard

INTERRUPTS:
a big performance problem I found was with ACPI interrupts,
find the problematic ones:

grep . -r /sys/firmware/acpi/interrupts/

if you an interrupt with thousands of calls, you can try to disable it,
BE AWARE THAT ANYTHING CAN HAPPEN

sudo echo “disable” > /sys/firmware/acpi/interrupts/gpeXX

source: https://wiki.archlinux.org/index.php/MacBook#kworker_using_high_CPU

add it as a systemd service:

nano /etc/systemd/system/suppress-gpeXX.service

add:

[Unit]
Description=Disables GPE XX, an interrupt that is going crazy on Macs

[Service]
ExecStart=/bin/bash -c ‘echo “disable” > /sys/firmware/acpi/interrupts/gpeXX’

[Install]
WantedBy=multi-user.target

enable it on boot:

sudo systemctl enable suppress-gpeXX

test it:

sudo systemctl start suppress-gpeXX

and check it has actually disabled the gpe:

cat /sys/firmware/acpi/interrupts/gpeXX

you should now read “disabled”,
it is better to test also a reboot.

POWER TUNING:

install powertop and run it:

sudo apt-get install powertop
sudo powertop –auto-tune

TODO: add it as a system service on startup

install TLP:

sudo apt-get install tlp tlp-rdw

source: http://linrunner.de/en/tlp/docs/tlp-linux-advanced-power-management.html#installation

GRUB TUNING:

follow: https://help.ubuntu.com/community/MacBookAir6-2/Trusty#Finetuning_Powersave_functionsTODO: add steps

BACKLIGHT and SUSPEND:

one issue is that the backlight does not automagically switch on after suspend/hibernate,
follow this guide: https://help.ubuntu.com/community/MacBookAir6-2/Trusty#Backlight

source: https://github.com/patjak/mba6x_bl

HIBERNATE:

sudo nano /etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla

add:

[Re-enable hibernate by default in upower]
Identity=unix-user:*
Action=org.freedesktop.upower.hibernate
ResultActive=yes

[Re-enable hibernate by default in logind]
Identity=unix-user:*
Action=org.freedesktop.login1.hibernate;org.freedesktop.login1.handle-hibernate-key;org.freedesktop.login1;org.freedesktop.login1.hibernate-multiple-sessions;org.freedesktop.login1.hibernate-ignore-inhibit
ResultActive=yes

source: https://help.ubuntu.com/16.04/ubuntu-help/power-hibernate.html

JAVA:

sudo apt install icedtea-8-plugin openjdk-8-jre

Install THUNAR file manager:

apt-get install thunar

xdg-mime default Thunar.desktop inode/directory

HOT CHANGING DISK IN MDADM RAID ARRAY

2016-04-30T00:00:00+00:00

to hot change a disk in an mdadm array, the disk to remove is sdb, the disk to add is sdc:

HAVE A BACKUP OF YOUR DATA
really,
backup your data

Verify the configuration of the array:

#cat /proc/mdstat

Clone the partition table on the news disk:
using sgdisk (install gdisk if you don’t have it) backup the partition table of the source disk:

sgdisk -b part.table /dev/sdb

the restore it on the destination disk, the -G switch is to generate new GUIDS for the partition since sgdisk also clones the GUIDS:

sgdisk -l part.table -G /dev/sdc

verify that the partitions are ok.

Modify the RAID array:

remove the “old” disk:

mdadm –manage /dev/md0 –fail /dev/sdb1
mdadm –manage /dev/md0 –remove /dev/sdb1

add the “new” disk to the array:

mdadm –manage /dev/md0 –add /dev/sdc1

Monitor the rebuild of the array:

#cat /proc/mdstat
[CUT]
[==>………………] recovery = 14% (nnnn/nnnnnnnn) finish=50.6min speed=57001K/sec
[CUT]

Wait for the rebuild.
Patiently.

Neural Paintings

2016-04-24T00:00:00+00:00

source: http://www.makeuseof.com/tag/create-neural-paintings-deepstyle-ubuntu/

sudo apt-get install git lua5.2 luarocks luajit libprotobuf-dev protobuf-compiler
curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash

test torch:
luajit -ltorch

install loadcaffe:

sudo luarocks install loadcaffe

if you have an old version of gcc you have to (source):

git clone https://github.com/szagoruyko/loadcaffe
cd loadcaffe
nano CMakeLists.txt
change the line: add_definitions(-std=c++11) to: add_definitions(-std=c++0x)
luarocks make

then:

sudo luarocks install image
sudo luarocks install nn

to install cuda support:

sudo luarocks install cutorch
sudo luarocks install cunn

but since the max allocable memory is the max memory of your gpu it is best to use cudnn, you have to register here: https://developer.nvidia.com/cudnn, then download the package and install (source):

tar -xvzf cudnn-7.0-linux-x64-v4.0-prod.tgz
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/cudnn.h /usr/local/cuda/include/
sudo luarocks install cudnn

and test it by running:

th neural_style.lua -gpu 0 -backend cudnn

if you have problems running it and it gives the error (source):

[CUT] ‘libcudnn (R4) not found in library path.
Please install CuDNN from https://developer.nvidia.com/cuDNN
Then make sure files named as libcudnn.so.4 or libcudnn.4.dylib are placed in your library load path (for example /usr/local/lib , or manually add a path to LD_LIBRARY_PATH)

you have to set the library path correctly:

export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH

now we can create our work folder and clone the git repo of the neuralnetwork,
it’s nice to do that were we a bit of space since it will use about ~600MB:

sudo git clone https://github.com/jcjohnson/neural-style.git
cd neural-style

now we can download the model, it will take a bit:

sudo sh models/download_models.sh

Images generated with simpler nerual network (nin_imagenet_conv):

Images generated with neural-style:

Notes:
using a smaller network: http://liipetti.net/erratic/2016/03/21/using-nin-imagenet-conv-in-neural-style/
ram discussion: https://github.com/jcjohnson/neural-style/issues/150
main repo: https://github.com/jcjohnson/neural-style
NIN_network: https://drive.google.com/folderview?id=0B0IedYUunOQINEFtUi1QNWVhVVU&usp=drive_web
cudnn: https://github.com/soumith/cudnn.torch
cunn issues: https://github.com/torch/cunn/issues/80
other issues: https://github.com/hughperkins/clnn/issues/18

NVIDIA CUDA LINUX HEADLESS

2016-02-26T00:00:00+00:00

Download the latest nvidia drivers for linux from Nvidia’s site and install it:

./NVIDIA-Linux-x86_64-xxx.xx.run

(I think I manually installed nvidia-smi)

To use nvidia-settings to do things like setting the fan speed you have to activate some parts of X, I have to re-document that part but I used infos from [0] and [1].

my rc.local:

Xorg :1 &
sleep 5
export DISPLAY=:1
nvidia-settings -a “[gpu:0]/GPUFanControlState=1”
exit 0

when you want to use nvidia-settings you have to execute in the shell you are using

export DISPLAY=:1

set fan speed (if you already set GPUFanControlState=1):

nvidia-settings -a “[fan:0]/GPUTargetFanSpeed=xx”

set power cap:

nvidia-smi –power-limit=xx

you can set the power capping dinamically, even when you have tasks using the cpu.

monitor card status:

nvidia-smi

[0] https://sites.google.com/site/akohlmey/random-hacks/nvidia-gpu-coolness

[1] https://devtalk.nvidia.com/default/topic/789888/set-fan-speed-without-an-x-server-solved-/

RUN AN OLD VERSION OF CUDAHASHCAT

2016-02-22T00:00:00+00:00

for benchmarking purposes I needed to run an old version cudaHashcat, the 1.31, now version 2.01 is out.

I got the error:

“ERROR: This copy of cudaHashcat is outdated. Get a more recent version.”

The solution is installing libfakedate:

apt-get install fakedate

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/faketime/libfaketime.so.1 FAKETIME=”-60d” /root/cudaHashcat-1.31/cudaHashcat64.bin –options-etcetera

note that the path of the lib should be the correct one for your system, you tune the fake date changing thedays back and forth, for me: “-60d”

source: https://serverfault.com/questions/138325/faking-the-date-for-a-specific-shell-session

NEXUS 9 "FAILED (REMOTE DATA LENGTH IS TOO LARGE)"

2016-01-23T00:00:00+00:00

if the flashing of the factory images from google fails try:

cd ~/Downloads
tar zxvf volantis-lmy47x-factory-3efdc8d4.tgz
cd volantis-lmy47x/
fastboot flash bootloader bootloader-flounder-xxxxxxxxx.img
fastboot reboot-bootloader
unzip image-volantis-lmy47x.zip
fastboot flash system system.img
fastboot flash recovery recovery.img
fastboot flash cache cache.img
fastboot flash boot boot.img
fastboot flash vendor vendor.img
select hboot and then factory reset (this take about 5 mintues).

source: https://www.reddit.com/r/Nexus9/comments/35ole1/anyone_else_having_problems_trying_to_flash_511/

Tinc switched on Debian 9 "stretch" (and old configs of debian 7)

2016-01-02T00:00:00+00:00

Install Tinc: #

apt-get install tinc

Configuration: #

the tinc configuration is based on some files:

/etc/tinc/
nets.boot this file contains the vpns that will be activated on boot (automatic creation)
vpn0/ this folder is a vpn and contains all the correlated files (manual creation)
rsa_key.priv this file is the private key of the local tinc host (automatic creation)
tinc.conf this file is the configuration of the vpn (manual creation)
tinc-up this file defines the behaviour of the interface on start (manual creation)
tinc-down this file defines the behaviour of the interface on stop (manual creation)
hosts/ this folder contains the public keys of the servers participating the vpn (manual creation)
- host01 this file is the public key of local host host01 (automatic creation)
- host02 this file is the public key of remote host host02 (manual transfer from host02)
- host03 like host02

to configure a tinc vpn in which all the hosts are “on the same switch” edit these files on the hosts

mkdir /etc/tinc/vpn0 #create the vpn main directory
cd /etc/tinc/vpn0 #
mkdir hosts #create the public key folder
touch tinc.conf tinc-up tinc-down #touch the files that will be edited later
chmod +x tinc-up tinc-down #make the interface files executable

then edit:
tinc.conf:

Name = host01                                      #name of the host, this must be the same as in the folder hosts
AddressFamily = ipv4                          #I live in the past, ipv4 is ok for me
Mode = switch                                      #the mode of the vpn, I choose switch
Device = /dev/net/tun                           #the device, I found that that parameter is ok like that (otherwise it won’t work)
ConnectTo = host01                             #the hosts to connect to on start (or every 15min as I read from the docs)
ConnectTo = host03                             #as above

tinc-up:

#!/bin/sh
ifconfig $INTERFACE 192.168.0.1 netmask 255.255.255.0                     #change 192.168.0.1 as the ip of the interface of the host (gor host01 I set 192.168.0.1, for host02 192.168.0.2)

tinc-down:

#!/bin/sh
ifconfig $INTERFACE down

and give these last files execution permissions:

chmod +x tinc-*

then you can execute the creation of the keys with the command:

tincd -n vpn0 -K4096

which will create the private (/etc/tinc/vpn0/rsa_key.priv) and public key (/etc/tinc/vpn0/rsa_key.pub) for the local host,
this step must be executed on all hosts.
then you have to add your hosts file for your own machine using the content of the public key:

in /etc/tinc/vpn0/hosts/yourmachinename

Address=98.263.34.11 655                          #the public ip address or FQDN (this changes for every host, obviously)
Port=655                                          #the port to connect
Compression=0                                     #the compression (I have not used it)
Subnet=192.168.0.1/32                             #your host's ip on tinc network

—–BEGIN RSA PUBLIC KEY—–
##########################################
—–END RSA PUBLIC KEY—–

then you have to copy this host file in the folder /etc/tinc/vpn0/hosts/ of the other hosts,
and viceversa, you have to copy the other hosts you want to connect to to your machines' folder.

Logging: #

edit the systemd unit file:

nano /lib/systemd/system/tinc@.service

and add "--logfile" ad the end of the Exec commands:

ExecStart=/usr/sbin/tincd -n %i -D --logfile
ExecReload=/usr/sbin/tincd -n %i -kHUP --logfile

Autostart: #

you can start your service using:

systemctl enable tinc@vpn

to start or restart:

systemctl start/restart tinc@vpn

LOGGING & DEBUGGING: #

if you start tinc with --logfile as above it will automatically create a file /var/log/tinc.vpn0.log
you can use the killsignals to output some useful stats on your logifle.
to do everything in one line I use:

vpn0_pid=$(cat /var/run/tinc.vpn0.pid) && kill -s USR1 $vpn0_pid && cat /var/log/tinc.vpn0.log

IPTABLES: #

you have to allow the port 655 in tcp and udp, if yout default is to dropeverything in INPUT, you have to allow the network in input:

-A INPUT -d PUBLIC_IP -i INTERFACE -p udp -m udp –dport 655 -j ACCEPT
-A INPUT -d PUBLIC_IP -i INTERFACE -p tcp -m tcp –dport 655 -j ACCEPT
-A INPUT -i vpn0 -j ACCEPT

!!! this part is only valid for debian 7: #

then you can start the vpn with

tincd -n vpn0 –logfile

and kill it with:

tincd -n vpn0 -k

to start automatically on boot the vpn add the vpn name in the file

/etc/tinc/nets.boot

if you append –logfile -d at your entry in nets.boot which might become something like:

vpn –logfile -d

you’ll find a neat /var/log/tinc.vpn.log logfile (won’t work in Debian9, see below)

main sources:
http://stacksetup.com/VPN/UsingTinc
https://silvenga.com/deploy-a-tinc-mesh-vpn-running-tap/
other sourced:
https://www.digitalocean.com/community/tutorials/how-to-install-tinc-and-set-up-a-basic-vpn-on-ubuntu-14-04
https://wiki.archlinux.org/index.php/Tinc

HUBICFUSE DEBIAN WHEEZY

2015-12-19T00:00:00+00:00

source: https://github.com/TurboGit/hubicfuse

in /etc/apt/sources.list add:

deb http://http.debian.net/debian wheezy-backports main

then:

apt-get update

apt-get install gcc make curl libfuse-dev pkg-config libcurl4-openssl-dev libxml2-dev libssl-dev libjson-c-dev libmagic-dev

then:

cd hubicfuse
./configure
make
make install

you can safely ignore make errors like:

cloudfsapi.c:474:7: warning: call to ‘_curl_easy_setopt_err_long’ declared with attribute warning: curl_easy_setopt expects a long argument for this option [enabled by default]
cloudfsapi.c:485:9: warning: call to ‘_curl_easy_setopt_err_long’ declared with attribute warning: curl_easy_setopt expects a long argument for this option [enabled by default]

to configure the client you have to create a new app in Myaccount –> Developers –> Add application, add a name and a URL (doesn’t need to be real).
Then click on Details of the newly crated app and use the “Client ID” and “Secret Client” for the next step.
In the hubicfuse folder execute the script:

./hubic_token

you will need to enter some info and to set the permission, in the end you will get:

Here is what your app needs to connect to hubiC: #

client_id=********
client_secret=********
refresh_token=********

copy this info in your ~/.hubicfuse

then mount:

hubicfuse /hubic -o noauto_cache,sync_read,allow_other

to automate the mount just add the same line you use from the cli to a file executed on startup like rc.local

WHAT DOES WORK:

zbackup

WHAT DOESN’T WORK:

renaming doesn’t work, moving to another folder works
concurrent use of the same folder from 2 different machines. I won’t recommend that.

ZBACKUP – VERY QUICK QUICKSTART

2015-12-08T00:00:00+00:00

source: http://zbackup.org/

install on debian:

apt-get install cmake libssl-dev libprotobuf-dev liblzma-dev liblzo2-dev zlib1g-dev protobuf-compiler protobuf-c-compiler

clone the repo:

git clone https://github.com/zbackup/zbackup

build & install:

cd zbackup
cmake .
make
sudo make install

or just run as ./zbackup #

initiate repo:

zbackup init –non-encrypted /zbackup/repo

backup something:

tar c foler_to_backup | zbackup backup –threads 2 –cache-size 1024mb –non-encrypted /zbackup/repo/backups/backup_name_timestamp

restore something:

zbackup restore –threads 2 –cache-size 1024mb –non-encrypted /data/zbackup/repo/backups/backup_sh-im01_201512040415 > backup_sh-im01_201512040415.tar

delete something:
delete the backup file from your /zbackup/repo/backups and run this command on the repo to perform garbage collection and delete unused chunks:

zbackup gc –non-encrypted /zbackup/repo/

notes:
adjust –threads and –cache-size as needed

mirror the repo to another server:
just copy it, I use rsync.
then restore the backup pointing to the backups folder like you would do on the main server.

APCUPSD ETHERNET SERVER CLIENT

2015-12-08T00:00:00+00:00

on server, /etc/apcupsd/apcupsd.conf:

UPSCABLE usb
UPSTYPE usb
DEVICE
NETSERVER on
NISIP 0.0.0.0 (or the net you want to serve)
NISPORT 3551

remember to open the port in iptables:

-A INPUT -i eth0 -p tcp -m tcp –dport 3551 -m comment –comment apcupsd -j ACCEPT

on client, /etc/apcupsd/apcupsd.conf:

UPSCABLE ether
UPSTYPE net
DEVICE x.x.x.x:3551

restart the service, to test the connection you can just issue:

#apcaccess

in the output you should see the lines:

UPSNAME : the same ups name on the server
CABLE : Ethernet Link
DRIVER : the same model on the server
SERIALNO : xxxxxxxxxxx

CACTI APC UPS

2015-12-07T00:00:00+00:00

followed the instructions on this post: http://www.evilbox.ro/linux/using-cacti-with-apc-ups-and-ubuntu/

but I had to modify the script hardcoding the server and port IP because it was not working:

#!/usr/bin/env bash
/sbin/apcaccess status x.x.x.x 3551 | /usr/share/cacti/site/scripts/apcupsd.pl

I created my graph template:
cacti_graph_template_apc_ups.xml

Here is the mirror of the original article in case something goes missing, the mirror of the original files is at the end:

USING CACTI WITH APC UPS AND UBUNTU
Linux

by Bogdan

Tested on :
Ubuntu 10.10 Server x64
Cacti 0.8.7g with NET-SNMP 5.x and RRDTool 1.4.x
APC UPS BR1500I with USB connection and apcupsd (3.14.8-2) package.
The original forum topic is here.
Cacti APC UPS image
You need to have apcupsd package installed, along with cacti.
Cacti can monitor my APC UPS using two scripts that rely on the information provided by apcaccess command.
The two scripts must be located in cacti’s scripts folder (in my case : /usr/share/cacti/site/scripts).

Changes : my UPS does not have an internal temperature sensor and cannot measure line frequency. So i modified the script to only show valid information (from apcaccess).
Stats monitored :

Time remaining in minutes
Battery charge level in %
Battery voltage in Volts
Utility line voltage in Volts
UPS load in %
My (customized and zipped) scripts are : apcupsd.pl and check_ups.sh and they must be executable.
Graph template (with dependencies, zip archive) is : cacti_graph_template_apc_ups_statistics
Data template is : cacti_data_template_apc_ups_statistics (included in graph template).

One change from the original data template was to increase the maximum value from 220 to 250 to the data source item line_volt, because the output in my case exceeds 220 (usually 230V) and the utility voltage was not shown.

Troubleshooting :

Be sure that the scripts are executable and the info in check_ups.sh is correct (the path for the check_ups.shscript)
Run check_ups.sh localhost 3551. It should display some statistics like :

line_volt:230.0 load_pct:7.0 b_charge:100.0 time_left:115.2 bat_volt:27.3

If it doesn’t, run the command apcaccess or apcaccess status. If you still get no output with ups stats, check to see if you have apcupsd package installed (see the beginning of the article for a link with install instructions).

Check RRD with command : rrdtool fetch /var/lib/cacti/rra/localhost_b_charge_117.rrd MAX where localhost_b_charge_117.rrd is the the data source file for Localhost – APC UPS Statistics. So look in Console -> Data Sources -> Localhost – APC UPS Statistics for the correct name and path.
Also check the Cacti log file, locate in Utilities -> System Utilites -> View Cacti Log file

cacti_data_template_apc_ups_statistics
check_ups
apcupsd
cacti_graph_template_apc_ups_statistics

Notes on Ubuntu 14.04 LTS

2015-10-10T00:00:00+00:00

Problem: apt-get update gives “Hash Sum mismatch”
create new list with http://repogen.simplylinux.ch/
edit /etc/apt/sources.list with the new list
sudo rm -rf /var/lib/apt/lists/partial/*
sudo rm -rf /var/lib/apt/lists/*
sudo rm /etc/apt/sources.list.save
sudo apt-get update

Persistent iptables:
sudo apt-get install iptables-persistent
to save the config: sudo invoke-rc.d iptables-persistent save

Enable hibernation:
https://help.ubuntu.com/14.04/ubuntu-help/power-hibernate.html

won’t shut down:
sudo nano /etc/default/grub
add “acpi=force” to the line “GRUB_CMDLINE_LINUX_DEFAULT=”
it might seem somthing like: GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash acpi=force”

INTEL NUC I5 AS VM SERVER

2014-05-27T00:00:00+00:00

After about a year with the Celeron 847 NUC (DCCP847DYE) with Debian and qemu I decided to upgrade the little one with another little one: the newer NUC i5-4250 (D54250WYK) and Debian 7, some notes:

it supports ONLY DDR3L, which means only 1.35v sodimms, so you cannot reuse the 1.5v ram you already have..
it has some issues with video outs, I had to disconnect, reboot and reconnect many times in order to obtain a signal on the monitor and after that if I switch the monitor input to another and get back to the NUC’s one I have to reboot to, hopefully, get the signal back..
I read that some “old” netinst of Debian hadn’t the drivers for the Gigabit ethernet interface so you had to boot an unstable to have the driver, I teste with netinst of Debian 7.5 (downloaded today 2014/05/27) and it has the drivers needed
I chose this version of the NUC instead the one with 2.5″ HDD support because this one was ready available and for the moment I could use the mSata disk I already have to boot the os.

UBUNTU IBM FLEX CMM AND IMM REMOTE CONTROL PROBLEMS

2013-06-19T00:00:00+00:00

On Ubuntu (12.10) there might be some problems using the java app of the remote console,
if so, from apt install libstdc++5.

i’m using openjdk6/7 with icedtea plugin.

solution found here: https://forums.oracle.com/message/9641538

PROJECT ARMADILLO NODE V0.0.1

2013-04-03T00:00:00+00:00

First release of the code of the Armadillo-Node

2013/03/04 version 0.0.1

features

read and display temp/hum of a DHT22 (or DHT21 or DHT11)
read/write some Digital I/O
read all 6 analog inputs
read/write 6 variables (2 used for the “no connection” warning)
“no connection” warning when the server doesn’t contatct the node for about 5 minutes

CODE:

/*
* WebServerParsing
* Respond to requests in the URL to change digital and analog output ports
* show the number of ports changed and the value of the analog input pins.
* for example:
* sending http://192.168.1.177/?pinD2=1 turns digital pin 2 on
* sending http://192.168.1.177/?pinD2=0 turns pin 2 off.
* This sketch demonstrates text parsing using the 1.0 Stream class.
*/

#include <SPI.h>
#include <Ethernet.h>
#include <DHT.h>

//*******************************************************************************************************
//DHT22
//*******************************************************************************************************
// Uncomment whatever type you’re using!
//#define DHTTYPE DHT11 // DHT 11
#define DHTTYPE DHT22 // DHT 22 (AM2302)
//#define DHTTYPE DHT21 // DHT 21 (AM2301)
#define DHTPIN 3 // what pin we’re connected to
DHT dht(DHTPIN, DHTTYPE);

//*******************************************************************************************************
//Ethernet Setup
//*******************************************************************************************************
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 172, 24, 33, 7 };
byte gateway[] = {172, 24, 33, 1 };
byte subnet[] = {255, 255, 255, 0 };

EthernetServer server(80);

int a = 0;
int b = 0;
int c = 0;
int d = 0;
long e = 0;
int f = 0;

int warn = 9; //pin for the warning led of no connection with the server
long timeout = 50000; //timeout in "program cycles", as of 2013/03/15 a cycle lasts about 0,006 secs, and every second about 166 cycles are executed

void setup()
{
Serial.begin(9600);
Ethernet.begin(mac, ip, gateway, subnet);
server.begin();
Serial.println("server ready");
dht.begin();
Serial.println("DHT ready");
pinMode(warn, OUTPUT);
}

void loop()
{
Serial.println(e);

if (e>0 & e<timeout & f!=1) {
digitalWrite(warn, HIGH);
}
if (e>0 & e<timeout & f==1) {
digitalWrite(warn, LOW);
}

if (e>timeout) {
f=99;
e=0;
}

e = e+1;

EthernetClient client = server.available();
if (client) {
while (client.connected()) {
if (client.available()) {
// counters to show the number of pin change requests
int digitalRequests = 0;
int analogRequests = 0;
if( client.find("GET /") ) { // search for ‘GET’
// find tokens starting with "pin" and stop on the first blank line
// search to the end of line for ‘pin’
while(client.findUntil("pin", "nr")){
char type = client.read(); // D or A
// the next ascii integer value in the stream is the pin
int pin = client.parseInt();
int val = client.parseInt(); // the integer after that is the value
if( type == ‘D’ & pin > 6 & pin < 9) {
Serial.print("Digital pin ");
pinMode(pin, OUTPUT);
digitalWrite(pin, val);
digitalRequests++;
}
else if( type == ‘A’){
Serial.print("Analog pin ");
analogWrite(pin, val);
analogRequests++;
}
else if( type == ‘V’ & pin > 19 & pin < 26){
Serial.print("Variable ");
switch (pin) {
case 20 : a = val; break;
case 21 : b = val; break;
case 22 : c = val; break;
case 23 : d = val; break;
case 24 : e = val; break;
case 25 : f = val; break;
}
}
else {
Serial.print("Unexpected type ");
Serial.print(type);
}
Serial.print(pin);
Serial.print("=");
Serial.println(val);
}
}
Serial.println();

// the findUntil has detected the blank line (a lf followed by cr)
// so the http request has ended and we can send a reply
// send a standard http response header
client.println("HTTP/1.1 200 OK");
client.println("Content-Type: text/html");
client.println();

// output the number of pins handled by the request
// client.print(digitalRequests);
// client.print(" digital pin(s) written");
// client.println("<br />");
// client.print(analogRequests);
// client.print(" analog pin(s) written");
// client.println("<br />");
// client.println("<br />");

float t = dht.readTemperature();
if (isnan(t) && (t) == 0.00) {
client.println("ERROR reading the sensor ");
}
else {
client.print("T ");
client.print(" = ");
client.print(t);
client.print(" *C");
client.println(" "); }

client.println("<br />");

float h = dht.readHumidity();
if (isnan(h)) {
client.println("ERROR reading the sensor ");
// client.print("DEBUG content of h ");
// client.print(h);
// client.println(" ");
}
else {
client.print("H ");
client.print(" = ");
client.print(h);
client.print(" %t");
client.println(" "); }

client.println("<br />");
// client.println("<br />");

// output the value of each analog input pin
for (int i = 0; i < 6; i++) {
client.print("A");
client.print(i);
client.print(" = ");
client.print(analogRead(i));
client.println("<br />");
}

client.println("<br />");

// output the value of digital pin 4 to 9
for (int i = 4; i < 10; i++) {
client.print("D");
client.print(i);
client.print(" = ");
client.print(digitalRead(i));
client.println("<br />");
}

client.println("<br />");

client.print("varA ");
client.print(" = ");
client.print(a);
client.println("<br />");
client.print("varB ");
client.print(" = ");
client.print(b);
client.println("<br />");
client.print("varC ");
client.print(" = ");
client.print(c);
client.println("<br />");
client.print("varD ");
client.print(" = ");
client.print(d);
client.println("<br />");
client.print("varE ");
client.print(" = ");
client.print(e);
client.println("<br />");
client.print("varF ");
client.print(" = ");
client.print(f);
client.println("<br />");

break;
}
}
// give the web browser time to receive the data
delay(1);
client.stop();
}
}

INTERFACING ARDUINO PRO MINI WITH THE ETHERNET SHIELD (W5100)

2013-03-14T00:00:00+00:00

http://proto-pic.com/Datasheets/Ethernet_PROMINI/Ethernet_PROMINI.pdf

source: proto-pic.com

RASPBERRYPI TRANSMISSION HEADLESS

2013-02-25T00:00:00+00:00

a nice mini guide: http://yetanothercomputingblog.blogspot.it/2013/01/building-headless-torrent-client-using.html

MULTIBOOT NEXUS 7 WITH MULTIROM

2013-02-21T00:00:00+00:00

Here’s a nice guide on how to install multiboot on your nexus 7:

http://www.youtube.com/watch?v=w-FRRzkhFiw

Reference XDA thread: http://forum.xda-developers.com/showthread.php?t=2011403

ROMs available: http://forum.xda-developers.com/showthread.php?t=2093797

I rooted my Nexus 7 using Nexus 7 Toolkit, you can obtain it here: http://forum.xda-developers.com/showthread.php?t=1809195
NB: It is worth to donate and receive the unlock code, it’s a fantastic software and donating gives you the ability to automatically update the program.

This also gives you the ability to test Palm’s (and then HP’s) WebOS

GOOGLE-AUTHENTICATOR ON SSH ON DEBIAN 6 (ON RASPBERRYPI)

2013-02-01T00:00:00+00:00

implementation of google-authenticator on Debian 6:

install the following packages:

sudo apt-get install libpam0g-dev

sudo apt-get install libreadline5-dev

and install the google-authenticator package:

wget https://google-authenticator.googlecode.com/files/libpam-google-authenticator-1.0-source.tar.bz2

tar -xjvf libpam-google-authenticator-1.0-source.tar.bz2

cd libpam-google-authenticator-1.0

make

sudo make install

then add “auth required pam_google_authenticator.so” at the begging of the file “/etc/pam.d/sshd”
sudo nano /etc/pam.d/sshd

and change the value “ChallengeResponseAuthentication” to “yes” in the file “/etc/ssh/sshd_config”
ChallengeResponseAuthentication yes

then run google-authenticator

and restart ssh: sudo service ssh restart

at your next login you’ll be asked for your Verification code:

sources:
for the procedure: http://www.mnxsolutions.com/security/two-factor-ssh-with-google-authenticator.html
for the packages to install: https://kb.askmonty.org/en/installing-correct-libraries-for-pam-and-readline/

SLOWMOVIDEO UNDER UBUNTU 12.10

2013-01-27T00:00:00+00:00

slowmoVideo can be found here: http://slowmovideo.granjow.net/

I downloaded and tried to install the 64bit Ubuntu 12.04 package,
“Ubuntu Software Center” warned me that a dependence could not be met: libglew1.5.

I had to download and install libglew1.5: https://launchpad.net/ubuntu/quantal/amd64/libglew1.5/1.5.7.is.1.5.2-1ubuntu4

Then the install of slowmoVideo went smoothly!

ARDUINO TELNET SERVER

2013-01-15T00:00:00+00:00

Found on this arduino forum post: http://arduino.cc/forum/index.php/topic,8533.0.html

the modified code to work with the updated Ethernet library:

[code lang=”arduino”]
/* *

Arduino Telnet Server *
7 July 2010 *
Basic Arduino I/O via *
a command line interface *
by Steve Lentz *
stlentz[at]gmail[dot]com *

Quick Start Instructions:

Set Ethernet address in code below.
Compile and upload sketch.
Connect Arduino to Ethernet.
Make sure link light is on.
Telnet to Arduino’s IP.
On some Telnet clients, hit return to wake up connection.
When connected, type ? for help.
Try a simple command such as ‘ar’.

Other notes
Tested on Duemilanove with Ethernet Shield.
Should work on compatible boards.
Tested with Win XP, OS X, and Debian Telnet clients.
Compiles to about 9 KB, can be made smaller by removing
unneeded commands, help message, etc.
I am an entirely self-taught C programmer; if you
don’t like my code, too bad ;-). /
// Ethernet parameters
//******************************************************************************************************
//Libraries
//*******************************************************************************************************
#include <Ethernet.h>
#include <SPI.h>
#include <DHT.h>

//*******************************************************************************************************
//DHT22
//*******************************************************************************************************
// Uncomment whatever type you’re using!
//#define DHTTYPE DHT11 // DHT 11
#define DHTTYPE DHT22 // DHT 22 (AM2302)
//#define DHTTYPE DHT21 // DHT 21 (AM2301)

#define DHTPIN 2 // what pin we’re connected to

DHT dht(DHTPIN, DHTTYPE);

//*******************************************************************************************************
//Ethernet Setup
//*******************************************************************************************************
byte mac[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
byte ip[] = {192, 168, 0, 10 };
byte gateway[] = {192, 168, 0, 254 };
byte subnet[] = {255, 255, 255, 0 };

//*******************************************************************************************************
// Other global variables
//*******************************************************************************************************
#define textBuffSize 9 //length of longest command string plus two spaces for CR + LF
char textBuff[textBuffSize]; //someplace to put received text
int charsReceived = 0;

boolean connectFlag = 0; //we’ll use a flag separate from client.connected
//so we can recognize when a new connection has been created
unsigned long timeOfLastActivity; //time in milliseconds of last activity
unsigned long allowedConnectTime = 300000; //five minutes

EthernetServer server(23); // Telnet listens on port 23
EthernetClient client = 0; // Client needs to have global scope so it can be called
// from functions outside of loop, but we don’t know
// what client is yet, so creating an empty object

void setup()
{
// setting pins 0 to 9 as outputs
// pins 10-13 are used by the Ethernet Shield
for(int i= 0; i < 10; i++) pinMode(i, OUTPUT);

Ethernet.begin(mac, ip, gateway, subnet);
server.begin();
dht.begin();
}

void loop()
{
// look to see if a new connection is created,
// print welcome message, set connected flag
if (server.available() && !connectFlag) {
connectFlag = 1;
client = server.available();
client.println("nSteve’s Arduino Telnet Server");
client.println("? for help");
printPrompt();
}

// check to see if text received
if (client.connected() && client.available()) getReceivedText();

// check to see if connection has timed out
if(connectFlag) checkConnectionTimeout();

// code to do other things in loop would go here

}
void printPrompt()
{
timeOfLastActivity = millis();
client.flush();
charsReceived = 0; //count of characters received
client.print("n>");
}
void checkConnectionTimeout()
{
if(millis() – timeOfLastActivity > allowedConnectTime) {
client.println();
client.println("Timeout disconnect.");
client.stop();
connectFlag = 0;
}
}
void getReceivedText()
{
char c;
int charsWaiting;

// copy waiting characters into textBuff
//until textBuff full, CR received, or no more characters
charsWaiting = client.available();
do {
c = client.read();
textBuff[charsReceived] = c;
charsReceived++;
charsWaiting–;
}
while(charsReceived <= textBuffSize && c != 0x0d && charsWaiting > 0);

//if CR found go look at received text and execute command
if(c == 0x0d) {
parseReceivedText();
// after completing command, print a new prompt
printPrompt();
}

// if textBuff full without reaching a CR, print an error message
if(charsReceived >= textBuffSize) {
client.println();
printErrorMessage();
printPrompt();
}
// if textBuff not full and no CR, do nothing else;
// go back to loop until more characters are received

}
void parseReceivedText()
{
// look at first character and decide what to do
switch (textBuff[0]) {
case ‘a’ : doAnalogCommand(); break;
case ‘d’ : doDigitalCommand(); break;
case ‘p’ : setPinMode(); break;
case ‘e’ : envCommand(); break;
case ‘c’ : checkCloseConnection(); break;
case ‘?’ : printHelpMessage(); break;
case 0x0d : break; //ignore a carriage return
default: printErrorMessage(); break;
}
}
void doDigitalCommand()
// if we got here, textBuff[0] = ‘d’
{
switch (textBuff[1]) {
case ‘r’ : readDigitalPins(); break;
case ‘w’ : writeDigitalPin(); break;
default: printErrorMessage(); break;
}
}
void readDigitalPins()
// if we got here, textBuff[0] = ‘d’ and textBuff[1] = ‘r’
{
int pin;
if (textBuff[2] == 0x0d) {
// output the valueof each digital pin
for (int i = 0; i < 10; i++) outputPinState(i);
}
else {
pin = parseDigit(textBuff[2]);
if(pin >=0 && pin <=9) outputPinState(pin);
else printErrorMessage();
}
}
void outputPinState(int pin)
{
client.print("digital pin ");
client.print(pin);
client.print(" is ");
if (digitalRead(pin)) {
client.println("HIGH");
}
else
client.println("LOW");
}
void writeDigitalPin()
// if we got here, textBuff[0] = ‘d’ and textBuff[1] = ‘w’
{
int pin = -1;
int pinSetting = -1;
if (textBuff[3] == ‘=’ && textBuff[6] == 0x0d) {
//if yes, get the pin number, setting, and set the pin
pin = parseDigit(textBuff[2]);
pinSetting = parsePinSetting();
if(pin > -1 && pinSetting == 0) {
digitalWrite(pin, LOW);
client.println("OK");
}
if(pin > -1 && pinSetting == 1) {
digitalWrite(pin, HIGH);
client.println("OK");
}
if(pin < 0 || pinSetting < 0) printErrorMessage();
}
else printErrorMessage();
}
int parsePinSetting()
//look in the text buffer to find the pin setting
//return -1 if not valid
{
int pinSetting = -1;
if(textBuff[4] == ‘l’ && textBuff[5] == ‘o’) pinSetting = 0;
if(textBuff[4] == ‘h’ && textBuff[5] == ‘i’) pinSetting = 1;
return pinSetting;
}
void doAnalogCommand()
// if we got here, textBuff[0] = ‘a’
{
switch (textBuff[1]) {
case ‘r’ : readAnalogPins(); break;
case ‘w’ : writeAnalogPin(); break;
default: printErrorMessage(); break;
}
}
void readAnalogPins()
// if we got here, textBuff[0] = ‘a’ and textBuff[1] = ‘r’
// check textBuff[2] is a CR then
// output the value of each analog input pin
{
if(textBuff[2] == 0x0d) {
for (int i = 0; i < 6; i++) {
client.print("analog input ");
client.print(i);
client.print(" is ");
client.println(analogRead(i));
}
}
else printErrorMessage();
}
void writeAnalogPin()
// if we got here, textBuff[0] = ‘a’ and textBuff[1] = ‘w’
{
int pin = -1;
int pwmSetting = -1;
if (textBuff[3] == ‘=’) {
//if yes, get the pin number, setting, and set the pin
pin = parseDigit(textBuff[2]);
if(pin == 3 || pin == 5 || pin == 6 || pin == 9) {
pwmSetting = parsepwmSetting();
if(pwmSetting >= 0 && pwmSetting <= 255) {
analogWrite(pin,pwmSetting);
client.println("OK");
}
else printErrorMessage();
}
else printErrorMessage();
}
else printErrorMessage();
}
int parsepwmSetting()
{
int pwmSetting = 0;
int textPosition = 4; //start at textBuff[4]
int digit;
do {
digit = parseDigit(textBuff[textPosition]); //look for a digit in textBuff
if (digit >= 0 && digit <=9) { //if digit found
pwmSetting = pwmSetting * 10 + digit; //shift previous result and add new digit
}
else pwmSetting = -1;
textPosition++; //go to the next position in textBuff
}
//if not at end of textBuff and not found a CR and not had an error, keep going
while(textPosition < 7 && textBuff[textPosition] != 0x0d && pwmSetting > -1);
//if value is not followed by a CR, return an error
if(textBuff[textPosition] != 0x0d) pwmSetting = -1;
return pwmSetting;
}
void setPinMode()
// if we got here, textBuff[0] = ‘p’
{
int pin = -1;
int pinModeSetting = -1;
if (textBuff[1] == ‘m’ && textBuff[3] == ‘=’ && textBuff[6] == 0x0d) {
//if yes, get the pin number, setting, and set the pin
pin = parseDigit(textBuff[2]);
pinModeSetting = parseModeSetting();
if(pin > -1 && pinModeSetting == 0) {
pinMode(pin, OUTPUT);
client.println("OK");
}
if(pin > -1 && pinModeSetting == 1) {
pinMode(pin, INPUT);
client.println("OK");
}
if(pin < 0 || pinModeSetting < 0) printErrorMessage();
}
else printErrorMessage();
}
int parseModeSetting()
//look in the text buffer to find the pin setting
//return -1 if not valid
{
int pinSetting = -1;
if(textBuff[4] == ‘o’ && textBuff[5] == ‘u’) pinSetting = 0;
if(textBuff[4] == ‘i’ && textBuff[5] == ‘n’) pinSetting = 1;
return pinSetting;
}
int parseDigit(char c)
{
int digit = -1;
digit = (int) c – 0x30; // subtracting 0x30 from ASCII code gives value
if(digit < 0 || digit > 9) digit = -1;
return digit;
}
void printErrorMessage()
{
client.println("Unrecognized command. ? for help.");
}
void checkCloseConnection()
// if we got here, textBuff[0] = ‘c’, check the next two
// characters to make sure the command is valid
{
if (textBuff[1] == ‘l’ && textBuff[2] == 0x0d)
closeConnection();
else
printErrorMessage();
}
void closeConnection()
{
client.println("nBye.n");
client.stop();
connectFlag = 0;
}
void printHelpMessage()
{
client.println("nExamples of supported commands:n");
client.println(" dr -digital read: returns state of digital pins 0 to 9");
client.println(" dr4 -digital read: returns state of pin 4 only");
client.println(" ar -analog read: returns all analog inputs");
client.println(" dw0=hi -digital write: turn pin 0 on valid pins are 0 to 9");
client.println(" dw0=lo -digital write: turn pin 0 off valid pins are 0 to 9");
client.println(" aw3=222 -analog write: set digital pin 3 to PWM value 222");
client.println(" allowed pins are 3,5,6,9");
client.println(" allowed PWM range 0 to 255");
client.println(" pm0=in -pin mode: set pin 0 to INPUT valid pins are 0 to 9");
client.println(" pm0=ou -pin mode: set pin 0 to OUTPUT valid pins are 0 to 9");
client.println(" et -environmental TEMP read: returns the state of the temp sensor");
client.println(" eh -environmental HUMidity read: returns the state of the humidity sensor");
client.println(" cl -close connection");
client.println(" ? -print this help message");
}

void envCommand()
// if we got here, textBuff[0] = ‘e’
{
switch (textBuff[1]) {
case ‘t’ : envTempCommand(); break;
case ‘h’ : envHumCommand(); break;
default: printErrorMessage(); break;
}
}

void envTempCommand()
// if we got here, textBuff[0] = ‘e’ and textBuff[1] = ‘t’
{
//float h = dht.readHumidity();
float t = dht.readTemperature();
// if (isnan(t)) {
if (isnan(t) && (t) == 0.00) {
client.println("ERROR reading the sensor ");
client.print("DEBUG content of t ");
client.print(t);
client.println(" "); }
else {
client.print("Temp: ");
client.print(t);
client.print(" *C");
client.println(" "); }
}

void envHumCommand()
// if we got here, textBuff[0] = ‘e’ and textBuff[1] = ‘h’
{
float h = dht.readHumidity();
if (isnan(h)) {
client.println("ERROR reading the sensor ");
client.print("DEBUG content of h ");
client.print(h);
client.println(" "); }
else {
client.print("Hum: ");
client.print(h);
client.print(" %t");
client.println(" "); }
}
[/code]

Powered by Debian stickers

2012-11-20T00:00:00+00:00

My Debian stickers finally arrived! I bought from LibreStickers, very fast shipping.
The stickers are very nice and seem pretty resistant and 10 cents per sticker are donated to the project!

Now my Debian 6 server is much nicer:

CISCO IPSEC VPN CLIENT – WIN7 – VODAFONE UMTS KEY – PROBLEMS

2012-11-15T00:00:00+00:00

solved following this tutorial: http://www.martinoroberto.it/en/problemi-tra-cisco-vpn-e-windows-7-64-bit-con-vodafone-key/

HP Smartarray P212 and Debian 6

2012-11-14T00:00:00+00:00

How to manage your fantastic RAID controller under Debian?

I was interested mainly in having the volumes and disks status to avoid data loss, you can use 2 tools:

cciss_vol_status

to install it under Debian simply do:

apt-get install cciss-vol-status

the tool monitors the status of the controller, like:

user@debian:~$ sudo cciss_vol_status /dev/cciss/c*d0
/dev/cciss/c0d0: (Smart Array P212) RAID 5 Volume 0 status: OK.

which is quite useful to see the status, possible statuses are:

  "OK." (0) - The logical drive is in good working order.

  "FAILED." (1) - The logical drive has failed,  and  no  i/o  to	it  is
  poosible.
    Additionally, failed drives will be identified by connector, box
    and bay, as well as vendor, model, serial number,  and  firmware
    revision.

  "Using interim recovery mode." (3) - One or more drives has failed,
    but  not	so  many that the logical drive can no longer operate.
    The failed drives should be replaced as soon as possible.

  "Ready for recovery operation." (4) -  Failed drive(s) have been
    replaced, and the controller is about to begin rebuilding redun
    dant parity data.

  "Currently recovering." (5) - Failed drive(s) have been replaced,
    and  the	controller  is	currently  rebuilding redundant parity
    information.

you can find the al the statutes and everything on the command here: http://cciss.sourceforge.net/cciss_vol_status.8.html
where I found all these useful informations? on this site: http://hwraid.le-vert.net/wiki/SmartArray
HP AcuCli
the other method is using HP’ ACU CLI for Debian (Array Controller Utility Command Line Interface),
you can find the last version fo ACUCLI for Debian on this HP’s FTP mirror: http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/debian/pool/non-free/
the last 64bit one is: hpacucli_8.70-8.0.2-2_amd64.deb, so you can download it to your server using wget:
wget http://downloads.linux.hp.com/SDR/repo/mcp/Debian/pool/non-free/hpacucli_9.40.1-1._amd64.deb
you can install the package following these steps (info found on this SITE), install dependencies:
#apt-get install lib32gcc1 lib32stdc++6
install your acucli .deb package
#dpkg -i hpacucli_8.70-8.0.2-2_amd64.deb
well done, now you can use the acucli to retrieve the system status or even configure and manage the controller,
to display the volumes and it’s status on the controller:
user@debian:~$ sudo hpacucli ctrl slot=1 logicaldrive all show status
logicaldrive 1 (5.5 TB, RAID 5): OK

to display the status of the disks:

user@debian:~$ sudo hpacucli ctrl slot=1 pd all show status

physicaldrive 1I:0:1 (port 1I:box 0:bay 1, 2 TB): OK
physicaldrive 1I:0:2 (port 1I:box 0:bay 2, 2 TB): OK
physicaldrive 1I:0:3 (port 1I:box 0:bay 3, 2 TB): OK
physicaldrive 1I:0:4 (port 1I:box 0:bay 4, 2 TB): OK

Where I found all these useful informations?

http://hwraid.le-vert.net/wiki/SmartArray
http://www.ganesh.me/337-install-hpacucli-on-ubuntu-debian.html
http://www.datadisk.co.uk/html_docs/redhat/hpacucli.htm
HP REPOSITORY FOR LINUX: http://downloads.linux.hp.com/SDR/repo/mcp/Debian/pool/non-free/

Building Microserver Nas with Debian 6 - part1

2012-11-14T00:00:00+00:00

After my Microserver with FreeNAS 8 crashed without chance of recovery I decided to build a Debian server and fileserver from scratch.

The hardware specs are:

Hp Microserver N36L with 8GB Ram (Corsair XMS3) [will use both the ram slots on the motherboard]
Boot from USB pen (Kingston Data Traveller G3 8GB) [will use the one and only internal USB port]
HP Smart Array P212/ZM, I added 256MB of cache but I don’t have the battery for the cache (BBWC) [will use one Pci-E 8x slot]
4x 2TB Drives (2x Hitachi + 2x Seagate) [will use all the 4 drive slots in your microserver]
optional second NIC connected, I used an HP NC110T [will use 1 Pci-E 1x]

Step 1 – Installing Debian

I simply used the Debian 6 netinstall CD that have both the 32bit and the 64bit installer downloadable HERE
I installed it on the USB pen without problems, keep in mind that having more than 1 drive in your machine force you to select the correct device to install the bootloader (grub), you can read the device from the partitioning tool where you select where to install Debian (in my case /dev/sdb).
I selected the packages “File Server”, “SSH Server” and something like “Common System Tools”.

Step 2 – Create the volume on the RAID Controller

Talking hardware: the HDD cage of your Microserver N36L and N40L will connect nicely to you P212 controller with the provided cable that was connected to the motherboard, I recommend to reroute the SAS cable like I did and suggested in this post: Fitting HP Smartarray P212 in Microserver

When your server boots, in the POST screen, you should see the Raid controller BIOS output, when asked press F8 to enter the controller configuration utility, then create your volume.
I created a RAID 5 volume with 4x 2TB drives.
NB: with the HP Smart Array P212 you’ll need some cache memory (minimum 256MB) to create a RAID 5 array, otherwise you’ll be forced to have only RAID 0,1,or 10 volumes and a maximum of 2 volumes.
WARNING: to improve drastically your performances you have to Enable the Cache, which is done automatically by the controller if you have the battery (BBWC) connected. Since I didn’t have a BBWC i decided to override this setting:

Reboot the server.
During POST, press the F8 key.
Select Cache Settings and press the Enter key.
Select Enable Write-Cache Battery Override and press the Enter key.
Press the F8 key, followed by the Enter key to continue.
Press the Esc key to exit.
Be warned that this is a very dangerous setting since in case of power loss you might corrupt the volumes managed by the controller.

I will follow this post with the Part 2 as soon as it’s ready!

Ubuntu on Google Nexus 7

2012-10-28T00:00:00+00:00

Followed this guide: https://wiki.ubuntu.com/Nexus7/Installation

Since I previously rooted the device using Google Nexus 7 ToolKit v3.2.0 I had to Re-Lock the device to make it pleasant to the Ubuntu installer.

I configured Gnome as my default desktop since I find unity laggy
My external keyboard with touchpad (Logitech K400) works perfectly atteched to the OTG cable
I still have to test many things
Sometimes the virtual keyboard doesn’t show up

but it’s quite funny,
recommended.

CISCO ASA 5505 8.2 HUB AND PIX 501 6.3 SPOKE EZVPN L2L VIA ASDM 6.4

2012-10-18T00:00:00+00:00

On the ASA, trough the ASDM,

select “Wizards”
then “IPsec VPN Wizard”
then select “Remote Access” mantaining “Enable inbound IPsec sessions etcetera..”, NEXT
first selection “Cisco VPN CLient….”, NEXT
select “Preshared Key” and write it, also te tunnel group name, in our example: preshared key: banana and group: grpBANANA, NEXT
use the local user DB, NEXT
if needed add the user (in our example: userBANANA), NEXT
select a pool from where it will take the IPs or create one, NEXT
select the DNS and the other stuff, NEXT
select interface “inside” and add in the exempt, I added the whole lan /24, if you want to tunnel ONLY the resources on the remote lan select “enable split tunneling…”, I HIGLY RECOMMEND TO SELECT SPLIT TUNNELING, NEXT
review your selections and then FINISH
NB: we’d like to use NEM (Network Extension Mode) so we’ll have to add a line manually to the config:

connect via ssh or serial to your magnificient ASA

locate you group policy lines (so you’ll look for the lines referring to “grpBANANA”
add “nem enable” to the configuration
yout config will look like this:

group-policy grpBANANA attributes
dns-server value IP_DNS_1 IP_DNS_2
vpn-tunnel-protocol IPSec
vsplit-tunnel-policy tunnelspecified
vsplit-tunnel-network-list value grpBANANA_splitTunnelAcl
vdefault-domain value banana.local
vnem enable

With the ASA config you should be good to go

PIX configuration:

I HIGHLY recommend to start with a clear pix configuration (use the command “configure factory-default” then remember to recreate the rsa keys: “hostname yourhosname”, “domain-name yourdomain”, “ca zeroize rsa”, “ca generate rsa key 1024”, “ca save all”, wr mem”)

command line via SSH or SERIAL, the final config will be like this:

PIX Version 6.3(5)
interface ethernet0 auto
interface ethernet1 100full
nameif ethernet0 outside security0
nameif ethernet1 inside security100
enable password xxxxxxxxxxxxxxxxxxxxxxxx encrypted
passwd xxxxxxxxxxxxxxxxxxxxxx encrypted
hostname bananaPIX
domain-name banana.local
fixup protocol dns maximum-length 512
fixup protocol ftp 21
fixup protocol h323 h225 1720
fixup protocol h323 ras 1718-1719
fixup protocol http 80
fixup protocol rsh 514
fixup protocol rtsp 554
fixup protocol sip 5060
fixup protocol sip udp 5060
fixup protocol skinny 2000
fixup protocol smtp 25
fixup protocol sqlnet 1521
fixup protocol tftp 69
names
pager lines 24
mtu outside 1500
mtu inside 1500
ip address outside dhcp setroute
ip address inside 192.168.0.0 255.255.255.0
ip audit info action alarm
ip audit attack action alarm
pdm logging informational 100
pdm history enable
arp timeout 14400
global (outside) 1 interface
nat (inside) 1 192.168.0.0 255.255.255.0 0 0
timeout xlate 0:05:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225 1:00:00
timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout sip-disconnect 0:02:00 sip-invite 0:03:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server TACACS+ max-failed-attempts 3
aaa-server TACACS+ deadtime 10
aaa-server RADIUS protocol radius
aaa-server RADIUS max-failed-attempts 3
aaa-server RADIUS deadtime 10
aaa-server LOCAL protocol local
no snmp-server location
no snmp-server contact
snmp-server community public
no snmp-server enable traps
floodguard enable
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 outside
ssh 0.0.0.0 0.0.0.0 inside
ssh timeout 5
console timeout 0
dhcpd address 192.168.0.1-192.168.0.50 inside
dhcpd lease 3600
dhcpd ping_timeout 750
dhcpd auto_config outside
dhcpd enable inside
vpnclient server PUBLIC_IP_ADDRESS_OF_ASA
vpnclient mode network-extension-mode
vpnclient vpngroup grpBANANA password ********
vpnclient username userBANANA password ********
vpnclient enable

this configuration is set to have the pix to search for DHCP on the outside interface so you can wire it to almost any network and be connected to your ASA
the really important part is this:

vpnclient server PUBLIC_IP_ADDRESS_OF_ASA
vpnclient mode network-extension-mode
vpnclient vpngroup grpBANANA password ********
vpnclient username userBANANA password ********
vpnclient enable

then you should be ready to go, remember to always try before needing something like this in a working state

Restore Dead Cisco ASA

2012-09-26T00:00:00+00:00

recover config http://www.gomjabbar.com/2011/05/16/removing-the-flash-memory-from-a-cisco-asa-5505/

Iomega Storcenter IX2-200 White light flashing - blinking light problem

2012-04-24T00:00:00+00:00

I had the problem of the ix2-200 Cloud Edition with the white light flashing continuosly like in this video (that is not one of my videos) http://www.youtube.com/watch?v=WS5PojOIznI

The steps I took:

Powered off
Disconnected 2nd drive
Powered on

white led blinking

powered off
Put in 2nd drive and disconnected 1st drive
powered on

white led blinking

off
disconnected both drives
on

white led blinking

off
pushed 15sec the power button
on

white led blinking

powered on
followed the procedure to factory default the device (pressing 15sec the reset button on the back)

waited, hurray!!!

after the blue light (disks) flashed and I heard some sounds of the disks spinning the white light became steady and the ix2 responded again!

Fitting HP Smartarray P212 in Microserver

2012-03-27T00:00:00+00:00

I managed to fit a spare HP P212 in the microserver,
to do so I had to cut some zip ties and re-route the SAS connector cable coming from the HDD cage:

pic of the re-routed sas cable descending near the fan:

pic of the cable connecting to the P212:

HP Microserver

2011-02-27T00:00:00+00:00

This is my Microserver N36L – E01

The HP Microserver N36L is a nifty piece of hardware, it’s very small and very well built and enginered .

It has 4 HDD slots and provides the rails to ease the access and substitution of the HDDs, the motherboard can slip of off the case when you unscrew the two thumb screws and disconnect some cables, so the 2 ram slots and the PCIe slots are easily serviceable.

The Microserver can be filled with a maximum of 8gb of ram making it perfect for building a ESX based home lab.
It supports perfectly ESXi 4.1 and 5, all the hardware is recognized and works without problems.

It’s form factor is pretty small and it’s very silent, it’s power consumption is much lower than a traditional server.

Pros:

small
silent
HDDs easily accessible
motherboard easily accessible
low power consumption

Cons:

limit of 8gb of ram
comes without cd reader
very small inside

Wrap up:
it’s perfect for the home lab, home server or if you want to build a NAS.