Actually replying to myself, with a solution to share

Joomla 4 use FontAwesome v5, and Joomla 5 FA v6. So I decided to go directly there!!

I followed the documentation here
https://docs.fontawesome.com/web/setup/host-yourself/webfonts

- Download the Free version 6.7.2 for the web https://fontawesome.com/download
- Inside the J51 template 'templates/<j51_template_name>', create the folder 'fontawesome-6' (this way, update or restoration of the template would not delete the new files, while keeping FA v4 from the template, used by the administrator side, for the template settings)
- From the downloaded FA pack, copy the following files:
sub-folder css
- brands.css
- brands.min.css
- fontawesome.css
- fontawesome.min.css
- regular.css
- regular.min.css
- solid.css
- solid.min.css
- v4-font-face.css
- v4-font-face.min.css
sub-folder webfonts
- fa-brands-400.ttf
- fa-brands-400.woff2
- fa-regular-400.ttf
- fa-regular-400.woff2
- fa-solid-900.ttf
- fa-solid-900.woff2
- fa-v4compatibility.ttf
- fa-v4compatibility.woff2

- finally, modify 'templates/<j51_template_name>/index.php' (hack, that would be overridden by a template restoration)

- replace the part, around line 199, after ' if ($fontawesome_sw) { '

    	$document->addStyleSheet('templates/' . $this->template . '/css/font-awesome.css');

by:

        // add the core fontawesome file v6

    	$document->addStyleSheet('templates/' . $this->template . '/fontawesome-6/css/fontawesome.css');

        // Your Version 4 js file gets replaced with the .css files for the styles you want to use

		$document->addStyleSheet('templates/' . $this->template . '/fontawesome-6/css/regular.css');

		$document->addStyleSheet('templates/' . $this->template . '/fontawesome-6/css/solid.css');

		$document->addStyleSheet('templates/' . $this->template . '/fontawesome-6/css/brands.css');

		// support v4 icon references/syntax

		$document->addStyleSheet('templates/' . $this->template . '/fontawesome-6/css/v4-font-face.css');

Then you should be able to display the new icons!!

I cannot use the outline version (regular style) for the moment, but this is not a big issue for me now.