New PO-based language translation support (#3471)

* lang: Add a PO language extractor with FW metadata support

Implement a straight-to-po language extractor which supports our custom
language requirements:

- _i/_I/ISTR for text string definitions
- _T for catalog translations (with back-reference support)
- //// EOL comments with:
  - MSG_ catalog entry name identifiers
  - c=X r=Y annotations for screen dimensioning checks
- Crude support for commented lines

All source locations are correctly referenced in the PO, with the
metadata colleted in the comment for further processing.

Several checks are implemented already during extraction:

- Correct catalog name assignment (no duplicates)
- Metadata checks for each entry

Further checks will be implemented by directly checking the translated PO file.

Requires "polib" and "regex" python modules.

* lang: Adapt lang-check to work directly on PO/POT files

* lang: Allow lang-extract to generate stable (pre-sorted) output directly

* lang: Further extend lang-extract consistency/error checking

- Do not parse inside preprocessor conditionals
- Distinguish between references and definitions
- Warn about missing references and definitions

* lang: lang-extract: warn about incorrect PROGMEM assignments

Check that ISTR is used along with PROGMEM_I1 in an attempt to spot
useless translated catalogs.

* lang: lang-extract: Improved handling of same-line translations

Correctly reference metadata on same-line translations.

* lang: lang-extract: Handle _O as a cat-ref

https://github.com/prusa3d/Prusa-Firmware/pull/3434

* lang: lang-extract: Warn about unused catalog definitions

* lang: lang-extract: Allow propagating translation comments via //

The definition:

    code //// definition [// comment]

will check [definition] as before, but blindly accumulate // comment.
The comment is then re-appended back into the PO files for translators
with the form:

    definition
    comment
    comment...

* lang: Fix incorrect display definitions

* lang: lang-extract: Check source encoding/charmap

* lang: Translate the degree symbol

* lang: Unbreak/cleanup DEBUG_SEC_LANG

* lang: Improve meaning of comment

* lang: Split charset conversions into an aux lib for future use

* lang: Implement lang-map.py to extract the translation symbol map

- Extracts the translatable symbol map for further use
- Computes a stable "language signature" from the map itself
- Optionally patches the binary update the symbols

* lang: Check for translation recoding problems

* lang: Implement a transliteration map to post-process translations

TRANS_CHARS is now used to replace unavailable symbols to the source
encoding, only while producing the language catalog.

* lang: Handle/check character replacements in lang-check

Filter the translation through TRANS_CHARS, so that the preview and
length check are performed correctly for expanding replacements such as
'ß' to 'ss'.

* lang: Implement lang-build.py to generate the final language catalog

* Cleanup .gitignore

* lang: Drop txt language files

* lang: Remove outdated translation scripts and obsolete docs

* lang: Update build scripts for new infrastructure

* lang: [no] Integrate accents from po/new/no.po

We now support accents natively

* lang: Remove redundant directory po/new/

* lang: Fix encoding of LCD characters in PO files

* lang: [hr] Fix wrapping in MSG_CRASH_DET_ONLY_IN_NORMAL

* lang: Sort and reformat PO files for further massaging

* lang: Switch to developer (dot) comments for PO metadata

* lang: Allow the IGNORE annotation to skip extraction

* lang: Fix missing/broken language metadata in sources

* lang: Add update-pot.sh and regenerate po/Firmware.pot

* lang: Add update-po.sh and refresh all PO files

* lang: Add summary documentation about the new translation workflow

* Add more ignored files

* CI: Add new required dependencies to travis

* lang: lang-build: Improve warning message

"referenced" was really meaning that data is being duplicated.

* lang: Respect the language order as defined in config.sh

This correctly splits normal and community-made entries during language
selection.

* lang: More typos in the documentation

* lang: Check for the maximum size of each language

Each table needs to fit within LANG_SIZE_RESERVED

* lang: Properly align _SEC_LANG to page boundaries

... instead of relying on _SEC_LANG_TABLE to calculate the offset

* lang: Build support for dual-language hex files

Detect the printer type by checking the current variant type.

On printers with no xflash (MK2*), generate one hex file for each
additional language file by patching the built-in secondary language
table during the build process

* lang: Mention lang-patchsec.py

* lang: Use color() instead of tput for clarity

* lang: Allow disabling terminal colors with NO_COLOR/TERM=dumb

* lang: Consistent use of redirection in config.sh

* lang: Stricter variant-type check for xflash support

* lang: Output size stats when building double-language hex files

* lang: Respect NO_COLOR in lang-check.py

* lang: Check for repeated/incorrect annotations

Catch errors such as "c=1 c=2"

* lang: Correct MSG_SLIGHT_SKEW/MSG_SEVERE_SKEW annotations

* lang: [it] Improve MSG_*_SKEW translation

* lang: Use INTLHEX instead of OUTHEX_P/S for configuration

We already have OUTHEX which is the compiled firmware.

Use INTLHEX for the final internationalized firmware, which is less
confusing. Also, assume it being a prefix for all generated hex
files, which reduces the number of variables set.

* lang: Move lang_map to lib.io for further use

* lang: lang-check: Accept a firmware map file to suppress unused string warnings

* lang: Use the map file to reduce useless warnings during fw-build

* lang: lang-check: Also suppress unused empty annotations

* lang: Fix MSG_MOVE_CARRIAGE_TO_THE_TOP_Z annotation

Refresh pot file

* lang: lang-check: Do not warn about same-word translations by default

Do not warn when one-word translations such as "No" result in "No" also
in other languages, since this is common in latin languages.

Allow to re-enable the warning with --warn-same

* lang: lang-build: Handle same-source/translation efficiently

* lang: [it] Explicitly add On/Off/Reset/Wizard to suppress warnings

Instead of displaying a warning, supress the warning and explicitly
translate each entry using english (which is the common/acceptable
word in these cases).

* lang: [it] Suppress more warnings

* lang: lang-check: Add intermediate "suggest" warning category

Warnings in the "suggest" category as shown as [S] as based on pure
speculation from the checking tool, such as the translation being
significantly shorter than the original.

As a result, they can be suppressed with --no-suggest

* lang: Return translation status from lang-check

- 0 if the translation only contains suggestions
- 1 if the translation contains warnings or errors

Check for the exit status in fw-build.sh, but do nothing at the moment
except printing a non-fatal error.

* lang: Remove "trim_trailing_whitespace=false" for po files

PO files got cleaned up/rewritten. We can now ensure they stay
consistent.

* lang: [sv] Re-integrate changes from 70c73cb

* lang: [no] Reintegrate changes from @pkg2000

.editorconfig

@@ -15,4 +15,3 @@ max_line_length             = 100
 
 [lang/po/*.po]
 end_of_line                 = crlf
-trim_trailing_whitespace    = false

.gitignore

@@ -1,74 +1,19 @@
-.settings
+# Temporary configuration
-.project
+/Firmware/Configuration_prusa.h
-.cproject
+
-Debug
+# Temporary language files
-__pycache__
+/lang/po/*.mo
-Firmware/Configuration_prusa.h
+/lang/tmp/
-Firmware/Doc
+/lang/Firmware-intl.hex
-/Firmware/.vs/Firmware/v14
+/lang/Firmware-intl-en_*.hex
-/Firmware/__vm
+
-/Firmware/Firmware.sln
+# Temporary files and directories
-/Firmware/Firmware.vcxproj
+*[~#]
-/Firmware/Firmware.vcxproj.filters
+*.tmp
-/Firmware/Firmware - Shortcut.lnk
+*.bak
-/Firmware/variants/1_75mm_MK3-MMU-EINSy10a-E3Dv6full.h.bak
-/Firmware/Marlin_main.cpp~RF12cfae7.TMP
-/Firmware/variants/1_75mm_MK3-EINSy10a-E3Dv6full.h.bak
-/html
-/latex
-/Doxyfile
-/Firmware/builds/1_75mm_MK3-EINY04-E3Dv6full
-/Firmware/Configuration_prusa.h.bak
-/Firmware/Configuration_prusa_backup.h
-/Firmware/ultralcd_implementation_hitachi_HD44780.h.bak
-/Firmware/ultralcd.cpp.bak
-/Firmware/temperature.cpp.bak
-/Firmware/pins.h.bak
-/Firmware/Marlin_main.cpp.bak
-/Firmware/language_pl.h.bak
-/Firmware/language_it.h.bak
-/Firmware/language_es.h.bak
-/Firmware/language_en.h.bak
-/Firmware/language_de.h.bak
-/Firmware/language_cz.h.bak
-/Firmware/variants/1_75mm_MK2-MultiMaterial-RAMBo13a-E3Dv6full.h
-/Firmware/variants/1_75mm_MK2-MultiMaterial-RAMBo10a-E3Dv6full.h
-/Firmware/variants/1_75mm_MK2-EINY01-E3Dv6full.h.bak
-/Firmware/variants/1_75mm_MK1-RAMBo13a-E3Dv6full.h
-/Firmware/variants/1_75mm_MK1-RAMBo10a-E3Dv6full.h
-/lang/*.bin
-/lang/*.hex
-/lang/*.dat
-/lang/*.tmp
-/lang/*.out
-/lang/not_tran*.txt
-/lang/not_used*.txt
-/lang/progmem1.chr
-/lang/progmem1.lss
-/lang/progmem1.txt
-/lang/progmem1.var
-/lang/text.sym
-/lang/textaddr.txt
-/build-env/
-/Firmware/Firmware.vcxproj
-/Firmware/Configuration_prusa_bckp.h
-/Firmware/variants/printers.h
-Configuration.tmp
-config.tmp
-/lang/lang_en.max
-/lang/po/new/*_new.po
-/lang/po/new/*_filered.po
-/lang/po/new/nonascii.txt
-/lang/po/new/lang_en*.txt
-/lang/po/new/output-*.txt
-/lang/po/new/*.mo
 .DS_Store
-**/.DS_Store
+__pycache__
-Firmware/.DS_Store
+
-Firmware/variant/.DS_Store
+# Generated files
-lang/.DS_Store
+/build-env/
-lang/po/.DS_Store
+/Firmware/Doc/
-lang/po/new/.DS_Store
-Tests/.DS_Store
-tools/.DS_Store
-tools/lib/.DS_Store

.travis.yml

@@ -1,6 +1,6 @@
 dist: focal
 before_install:
-  - sudo apt-get install -y ninja-build
+  - sudo apt-get install -y ninja-build python3-polib python3-pyelftools
   # Arduino IDE adds a lot of noise caused by network traffic, trying to firewall it off
   - sudo iptables -P INPUT DROP
   - sudo iptables -P FORWARD DROP

Firmware/Marlin_main.cpp

@@ -1168,23 +1168,6 @@ void setup()
 	uint32_t src_addr = 0x00000;
 	if (lang_get_header(1, &header, &src_addr))
 	{
-//this is comparsion of some printing-methods regarding to flash space usage and code size/readability
-#define LT_PRINT_TEST 2
-//  flash usage
-//  total   p.test
-//0 252718  t+c  text code
-//1 253142  424  170  254
-//2 253040  322  164  158
-//3 253248  530  135  395
-#if (LT_PRINT_TEST==1) //not optimized printf
-		printf_P(_n(" _src_addr = 0x%08lx\n"), src_addr);
-		printf_P(_n(" _lt_magic = 0x%08lx %S\n"), header.magic, (header.magic==LANG_MAGIC)?_n("OK"):_n("NA"));
-		printf_P(_n(" _lt_size  = 0x%04x (%d)\n"), header.size, header.size);
-		printf_P(_n(" _lt_count = 0x%04x (%d)\n"), header.count, header.count);
-		printf_P(_n(" _lt_chsum = 0x%04x\n"), header.checksum);
-		printf_P(_n(" _lt_code  = 0x%04x (%c%c)\n"), header.code, header.code >> 8, header.code & 0xff);
-		printf_P(_n(" _lt_sign = 0x%08lx\n"), header.signature);
-#elif (LT_PRINT_TEST==2) //optimized printf
 		printf_P(
 		 _n(
 		  " _src_addr = 0x%08lx\n"
@@ -1203,34 +1186,6 @@ void setup()
 		 header.code, header.code >> 8, header.code & 0xff,
 		 header.signature
 		);
-#elif (LT_PRINT_TEST==3) //arduino print/println (leading zeros not solved)
-		MYSERIAL.print(" _src_addr = 0x");
-		MYSERIAL.println(src_addr, 16);
-		MYSERIAL.print(" _lt_magic = 0x");
-		MYSERIAL.print(header.magic, 16);
-		MYSERIAL.println((header.magic==LANG_MAGIC)?" OK":" NA");
-		MYSERIAL.print(" _lt_size  = 0x");
-		MYSERIAL.print(header.size, 16);
-		MYSERIAL.print(" (");
-		MYSERIAL.print(header.size, 10);
-		MYSERIAL.println(")");
-		MYSERIAL.print(" _lt_count = 0x");
-		MYSERIAL.print(header.count, 16);
-		MYSERIAL.print(" (");
-		MYSERIAL.print(header.count, 10);
-		MYSERIAL.println(")");
-		MYSERIAL.print(" _lt_chsum = 0x");
-		MYSERIAL.println(header.checksum, 16);
-		MYSERIAL.print(" _lt_code  = 0x");
-		MYSERIAL.print(header.code, 16);
-		MYSERIAL.print(" (");
-		MYSERIAL.print((char)(header.code >> 8), 0);
-		MYSERIAL.print((char)(header.code & 0xff), 0);
-		MYSERIAL.println(")");
-		MYSERIAL.print(" _lt_resv1 = 0x");
-		MYSERIAL.println(header.signature, 16);
-#endif //(LT_PRINT_TEST==)
-#undef LT_PRINT_TEST
 
 #if 0
 		xflash_rd_data(0x25ba, (uint8_t*)&block_buffer, 1024);
@@ -1249,9 +1204,9 @@ void setup()
 		printf_P(_n("_SEC_LANG_TABLE checksum = %04x\n"), sum);
 		sum = (sum >> 8) | ((sum & 0xff) << 8); //swap bytes
 		if (sum == header.checksum)
-			puts_P(_n("Checksum OK"), sum);
+			puts_P(_n("Checksum OK"));
 		else
-			puts_P(_n("Checksum NG"), sum);
+			puts_P(_n("Checksum NG"));
 	}
 	else
 		puts_P(_n("lang_get_header failed!"));
@@ -1534,11 +1489,9 @@ void setup()
 		lcd_language();
 
 #ifdef DEBUG_SEC_LANG
-
 	uint16_t sec_lang_code = lang_get_code(1);
 	uint16_t ui = _SEC_LANG_TABLE; //table pointer
 	printf_P(_n("lang_selected=%d\nlang_table=0x%04x\nSEC_LANG_CODE=0x%04x (%c%c)\n"), lang_selected, ui, sec_lang_code, sec_lang_code >> 8, sec_lang_code & 0xff);
-
 	lang_print_sec_lang(uartout);
 #endif //DEBUG_SEC_LANG
 
@@ -3308,7 +3261,7 @@ static void gcode_G80()
         Sound_MakeSound(e_SOUND_TYPE_StandardAlert);
         bool bState;
         do   {                             // repeat until Z-leveling o.k.
-            lcd_display_message_fullscreen_P(_i("Some problem encountered, Z-leveling enforced ..."));
+            lcd_display_message_fullscreen_P(_i("Some problem encountered, Z-leveling enforced ...")); ////MSG_ZLEVELING_ENFORCED c=20 r=4
 #ifdef TMC2130
             lcd_wait_for_click_delay(MSG_BED_LEVELING_FAILED_TIMEOUT);
             calibrate_z_auto();           // Z-leveling (X-assembly stay up!!!)
@@ -3794,8 +3747,9 @@ static void gcode_M600(bool automatic, float x_position, float y_position, float
     if (!mmu_enabled)
     {
         KEEPALIVE_STATE(PAUSED_FOR_USER);
-        lcd_change_fil_state = lcd_show_fullscreen_message_yes_no_and_wait_P(_i("Was filament unload successful?"),
+        lcd_change_fil_state = lcd_show_fullscreen_message_yes_no_and_wait_P(
-                false, true); ////MSG_UNLOAD_SUCCESSFUL c=20 r=2
+                _i("Was filament unload successful?"), ////MSG_UNLOAD_SUCCESSFUL c=20 r=2
+                false, true);
         if (lcd_change_fil_state == 0)
         {
 			lcd_clear();
@@ -5348,7 +5302,7 @@ if(eSoundMode!=e_SOUND_MODE_SILENT)
         if (calibration_status() >= CALIBRATION_STATUS_XYZ_CALIBRATION) {
             //we need to know accurate position of first calibration point
             //if xyz calibration was not performed yet, interrupt temperature calibration and inform user that xyz cal. is needed
-            lcd_show_fullscreen_message_and_wait_P(_i("Please run XYZ calibration first."));
+            lcd_show_fullscreen_message_and_wait_P(_i("Please run XYZ calibration first.")); ////MSG_RUN_XYZ c=20 r=4
             break;
         }
 

Firmware/cardreader.cpp

@@ -877,7 +877,10 @@ void CardReader::presort() {
 				uint16_t counter = 0;
 				uint16_t total = 0;
 				for (uint16_t i = sortCountFiles/2; i > 0; i /= 2) total += sortCountFiles - i; //total runs for progress bar
-				menu_progressbar_init(total, (runs == 0)?_i("Sorting files"):_i("Sorting folders"));
+				menu_progressbar_init(
+                        total, (runs == 0)
+                        ? _i("Sorting files") ////MSG_SORTING_FILES c=20
+                        : _i("Sorting folders")); ////MSG_SORTING_FOLDERS c=20
 				
 				for (uint16_t gap = sortCountFiles/2; gap > 0; gap /= 2)
 				{

Firmware/config.h

@@ -60,6 +60,10 @@
 
 #define LANG_SIZE_RESERVED     0x3000 // reserved space for secondary language (12288 bytes). Maximum 32768 bytes
 
+#if (LANG_SIZE_RESERVED % 256)
+  #error "LANG_SIZE_RESERVED should be a multiple of a page size"
+#endif
+
 //Community language support
 #define COMMUNITY_LANG_GROUP 1
 
@@ -81,6 +85,7 @@
 #if (COMMUNITY_LANG_GROUP >=1 )
 #define COMMUNITY_LANGUAGE_SUPPORT
 #endif
+
 // Sanity checks for correct configuration of XFLASH_DUMP options
 #if defined(XFLASH_DUMP) && !defined(XFLASH)
 #error "XFLASH_DUMP requires XFLASH support"

Firmware/language.c

@@ -28,7 +28,7 @@ uint8_t lang_is_selected(void) { return 1; }
 #else //(LANG_MODE == 0) //secondary languages in progmem or xflash
 
 //reserved xx kbytes for secondary language table
-const char _SEC_LANG[LANG_SIZE_RESERVED] PROGMEM_I2 = "_SEC_LANG";
+const char __attribute__((aligned(256))) _SEC_LANG[LANG_SIZE_RESERVED] PROGMEM_I2 = "_SEC_LANG";
 
 //primary language signature
 const uint32_t _PRI_LANG_SIGNATURE[1] __attribute__((section(".progmem0"))) = {0xffffffff};
@@ -41,7 +41,7 @@ const char* lang_get_translation(const char* s)
 	if (lang_selected == 0) return s + 2; //primary language selected, return orig. str.
 	if (lang_table == 0) return s + 2; //sec. lang table not found, return orig. str.
 	uint16_t ui = pgm_read_word(((uint16_t*)s)); //read string id
-	if (ui == 0xffff) return s + 2; //translation not found, return orig. str.
+	if (ui == 0xffff) return s + 2; //id not assigned, return orig. str.
 	ui = pgm_read_word(((uint16_t*)(((char*)lang_table + 16 + ui*2)))); //read relative offset
 	if (pgm_read_byte(((uint8_t*)((char*)lang_table + ui))) == 0) //read first character
 		return s + 2;//zero length string == not translated, return orig. str.

Firmware/language.h

@@ -145,9 +145,7 @@ extern uint8_t lang_selected;
 #if (LANG_MODE != 0)
 extern const char _SEC_LANG[LANG_SIZE_RESERVED];
 extern const char* lang_get_translation(const char* s);
-/** @def _SEC_LANG_TABLE
+#define _SEC_LANG_TABLE ((uint16_t)&_SEC_LANG)
- *  @brief Align table to start of 256 byte page */
-#define _SEC_LANG_TABLE ((((uint16_t)&_SEC_LANG) + 0x00ff) & 0xff00)
 #endif //(LANG_MODE != 0)
 
 /** @brief selects language, eeprom is updated in case of success */

Firmware/messages.c

@@ -8,7 +8,7 @@
 #include "Configuration_prusa.h"
 
 //internationalized messages
-const char MSG_AUTO_HOME[] PROGMEM_I1 = ISTR("Auto home"); ////MSG_AUTO_HOMEc=18
+const char MSG_AUTO_HOME[] PROGMEM_I1 = ISTR("Auto home"); ////MSG_AUTO_HOME c=18
 const char MSG_BABYSTEP_Z[] PROGMEM_I1 = ISTR("Live adjust Z"); ////MSG_BABYSTEP_Z c=18
 const char MSG_BABYSTEP_Z_NOT_SET[] PROGMEM_I1 = ISTR("Distance between tip of the nozzle and the bed surface has not been set yet. Please follow the manual, chapter First steps, section First layer calibration."); ////MSG_BABYSTEP_Z_NOT_SET c=20 r=12
 const char MSG_BED[] PROGMEM_I1 = ISTR("Bed"); ////MSG_BED c=13
@@ -162,16 +162,16 @@ const char MSG_DIM[] PROGMEM_I1 = ISTR("Dim"); ////MSG_DIM c=6
 const char MSG_AUTO[] PROGMEM_I1 = ISTR("Auto"); ////MSG_AUTO c=6
 #ifdef IR_SENSOR_ANALOG
 // Beware - the space at the beginning is necessary since it is reused in LCD menu items which are to be with a space
-const char MSG_IR_04_OR_NEWER[] PROGMEM_I1 = ISTR(" 0.4 or newer");////c=18
+const char MSG_IR_04_OR_NEWER[] PROGMEM_I1 = ISTR(" 0.4 or newer");////MSG_IR_04_OR_NEWER c=18
-const char MSG_IR_03_OR_OLDER[] PROGMEM_I1 = ISTR(" 0.3 or older");////c=18
+const char MSG_IR_03_OR_OLDER[] PROGMEM_I1 = ISTR(" 0.3 or older");////MSG_IR_03_OR_OLDER c=18
-const char MSG_IR_UNKNOWN[] PROGMEM_I1 = ISTR("unknown state");////c=18
+const char MSG_IR_UNKNOWN[] PROGMEM_I1 = ISTR("unknown state");////MSG_IR_UNKNOWN c=18
 #endif
 
 //not internationalized messages
-const char MSG_AUTO_DEPLETE[] PROGMEM_N1 = ISTR("SpoolJoin"); ////c=13
+const char MSG_AUTO_DEPLETE[] PROGMEM_N1 = ISTR("SpoolJoin"); ////MSG_AUTO_DEPLETE c=13
-const char MSG_FIRMWARE[] PROGMEM_N1 = ISTR("Firmware"); ////c=8
+const char MSG_FIRMWARE[] PROGMEM_N1 = ISTR("Firmware"); ////MSG_FIRMWARE c=8
-const char MSG_TOSHIBA_FLASH_AIR_COMPATIBILITY[] PROGMEM_N1 = ISTR("FlashAir"); ////c=8
+const char MSG_TOSHIBA_FLASH_AIR_COMPATIBILITY[] PROGMEM_N1 = ISTR("FlashAir"); ////MSG_TOSHIBA_FLASH_AIR_COMPATIBILITY c=8
-const char MSG_PINDA[] PROGMEM_N1 = ISTR("PINDA");////c=5
+const char MSG_PINDA[] PROGMEM_N1 = ISTR("PINDA");////MSG_PINDA c=5
 const char WELCOME_MSG[] PROGMEM_N1 = ISTR(CUSTOM_MENDEL_NAME " OK."); ////c=20
 const char MSG_SD_WORKDIR_FAIL[] PROGMEM_N1 = "workDir open failed"; ////
 const char MSG_BROWNOUT_RESET[] PROGMEM_N1 = " Brown out Reset"; ////

Firmware/mmu.cpp

@@ -755,14 +755,14 @@ void manage_response(bool move_axes, bool turn_off_nozzle, uint8_t move)
 				lcd_clear();
 				setTargetHotend(hotend_temp_bckp, active_extruder);
 				if (((degTargetHotend(active_extruder) - degHotend(active_extruder)) > 5)) {
-					lcd_display_message_fullscreen_P(_i("MMU OK. Resuming temperature..."));
+					lcd_display_message_fullscreen_P(_i("MMU OK. Resuming temperature...")); ////MSG_MMU_OK_RESUMING_TEMPERATURE c=20 r=4
 					delay_keep_alive(3000);
 				}
 				mmu_wait_for_heater_blocking();
 			  }			  
 			  if (move_axes) {
 				  lcd_clear();
-				  lcd_display_message_fullscreen_P(_i("MMU OK. Resuming position..."));
+				  lcd_display_message_fullscreen_P(_i("MMU OK. Resuming position...")); ////MSG_MMU_OK_RESUMING_POSITION c=20 r=4
 				  current_position[X_AXIS] = x_position_bckp;
 				  current_position[Y_AXIS] = y_position_bckp;
 				  plan_buffer_line_curposXYZE(50);
@@ -773,7 +773,7 @@ void manage_response(bool move_axes, bool turn_off_nozzle, uint8_t move)
 			  }
 			  else {
 				  lcd_clear();
-				  lcd_display_message_fullscreen_P(_i("MMU OK. Resuming..."));
+				  lcd_display_message_fullscreen_P(_i("MMU OK. Resuming...")); ////MSG_MMU_OK_RESUMING c=20 r=4
 				  delay_keep_alive(1000); //delay just for showing MMU OK message for a while in case that there are no xyz movements
 			  }
 		  }
@@ -1338,7 +1338,7 @@ bool mmu_check_version()
 void mmu_show_warning()
 {
 	printf_P(PSTR("MMU2 firmware version invalid. Required version: build number %d or higher."), MMU_REQUIRED_FW_BUILDNR);
-	kill(_i("Please update firmware in your MMU2. Waiting for reset."));
+	kill(_i("Please update firmware in your MMU2. Waiting for reset.")); ////MSG_UPDATE_MMU2_FW c=20 r=4
 }
 
 void lcd_mmu_load_to_nozzle(uint8_t filament_nr)
@@ -1412,13 +1412,13 @@ bFilamentAction=false;                            // NOT in "mmu_fil_eject_menu(
 			{
 			    LcdUpdateDisabler disableLcdUpdate;
                 lcd_clear();
-                lcd_puts_at_P(0, 1, _i("Ejecting filament"));
+                lcd_puts_at_P(0, 1, _i("Ejecting filament")); ////MSG_EJECTING_FILAMENT c=20
                 mmu_filament_ramming();
                 mmu_command(MmuCmd::E0 + filament);
                 manage_response(false, false, MMU_UNLOAD_MOVE);
                 if (recover)
                 {
-                    lcd_show_fullscreen_message_and_wait_P(_i("Please remove filament and then press the knob."));
+                    lcd_show_fullscreen_message_and_wait_P(_i("Please remove filament and then press the knob.")); ////MSG_EJECT_REMOVE c=20 r=4
                     mmu_command(MmuCmd::R0);
                     manage_response(false, false);
                 }

Firmware/ultralcd.cpp

@@ -1580,7 +1580,7 @@ static void lcd_menu_fails_stats_print()
                       " %-7.7S X %-3d Y %-3d"),
                  _T(MSG_LAST_PRINT_FAILURES),
                  _T(MSG_POWER_FAILURES), power,
-                 _i("Runouts"), filam, fsensor_softfail, //MSG_RUNOUTS c=7
+                 _i("Runouts"), filam, fsensor_softfail, ////MSG_RUNOUTS c=7
                  _T(MSG_CRASH), crashX, crashY);
 #endif
     menu_back_if_clicked_fb();
@@ -1714,7 +1714,7 @@ static void lcd_menu_temperatures()
     lcd_menu_temperatures_line( _T(MSG_NOZZLE), (int)current_temperature[0] );
     lcd_menu_temperatures_line( _T(MSG_BED), (int)current_temperature_bed );
 #ifdef AMBIENT_THERMISTOR
-    lcd_menu_temperatures_line( _i("Ambient"), (int)current_temperature_ambient );  ////MSG_AMBIENT
+    lcd_menu_temperatures_line( _i("Ambient"), (int)current_temperature_ambient );  ////MSG_AMBIENT c=14
 #endif //AMBIENT_THERMISTOR
 #ifdef PINDA_THERMISTOR
     lcd_menu_temperatures_line( _T(MSG_PINDA), (int)current_temperature_pinda );  ////MSG_PINDA
@@ -2020,7 +2020,7 @@ static void lcd_support_menu()
   if (IP_address) {
       
       MENU_ITEM_BACK_P(STR_SEPARATOR);
-      MENU_ITEM_BACK_P(PSTR("Printer IP Addr:"));  ////MSG_PRINTER_IP c=18
+      MENU_ITEM_BACK_P(_i("Printer IP Addr:")); ////MSG_PRINTER_IP c=18
       MENU_ITEM_BACK_P(PSTR(" "));
       if (((menu_item - 1) == menu_line) && lcd_draw_update) {
           lcd_set_cursor(2, menu_row);
@@ -2858,10 +2858,10 @@ float _deg(float rad)
 //! 
 //! @code{.unparsed}
 //! |01234567890123456789|
-//! |Measured skew :0.00D|	MSG_MEASURED_SKEW c=14, c=4
+//! |Measured skew :0.00D|	MSG_MEASURED_SKEW c=14
 //! | --------------     |	STR_SEPARATOR
-//! |Slight skew   :0.12D|	MSG_SLIGHT_SKEW c=14, c=4
+//! |Slight skew   :0.12D|	MSG_SLIGHT_SKEW c=14
-//! |Severe skew   :0.25D|	MSG_SEVERE_SKEW c=14, c=4
+//! |Severe skew   :0.25D|	MSG_SEVERE_SKEW c=14
 //! ----------------------
 //! D - Degree sysmbol		LCD_STR_DEGREE
 //! @endcode
@@ -2878,8 +2878,8 @@ static void lcd_menu_xyz_skew()
 	 ),
 	 _i("Measured skew"),  ////MSG_MEASURED_SKEW c=14
 	 separator,
-	 _i("Slight skew"), _deg(bed_skew_angle_mild),  ////MSG_SLIGHT_SKEW c=14, c=4
+	 _i("Slight skew"), _deg(bed_skew_angle_mild),  ////MSG_SLIGHT_SKEW c=14
-	 _i("Severe skew"), _deg(bed_skew_angle_extreme)  ////MSG_SEVERE_SKEW c=14, c=4
+	 _i("Severe skew"), _deg(bed_skew_angle_extreme)  ////MSG_SEVERE_SKEW c=14
 	);
 	if (angleDiff < 100){
 		lcd_set_cursor(15,0);
@@ -3025,7 +3025,7 @@ static void lcd_babystep_z()
 	    lcd_set_cursor(0, 0);
 	    lcd_print(buffer.c);
 	    lcd_set_cursor(0, 1);
-		menu_draw_float13(_i("Adjusting Z:"), _md->babystepMemMMZ); ////MSG_BABYSTEPPING_Z c=15 Beware: must include the ':' as its last character
+		menu_draw_float13(_i("Adjusting Z:"), _md->babystepMemMMZ); ////MSG_BABYSTEPPING_Z c=15 // Beware: must include the ':' as its last character
 	}
 	if (LCD_CLICKED || menu_leaving)
 	{
@@ -3117,7 +3117,7 @@ void lcd_adjust_bed(void)
 //! 
 //! @code{.unparsed}
 //! |01234567890123456789|
-//! |Set temperature:    |	MSG_SET_TEMPERATURE c=20
+//! |Set temperature:    |
 //! |                    |
 //! | 210                |
 //! |                    |
@@ -3126,7 +3126,7 @@ void lcd_adjust_bed(void)
 void pid_extruder()
 {
 	lcd_clear();
-	lcd_puts_at_P(0, 0, _i("Set temperature:"));////MSG_SET_TEMPERATURE
+	lcd_puts_at_P(0, 0, _i("Set temperature:"));////MSG_SET_TEMPERATURE c=20
 	pid_temp += int(lcd_encoder);
 	if (pid_temp > HEATER_0_MAXTEMP) pid_temp = HEATER_0_MAXTEMP;
 	if (pid_temp < HEATER_0_MINTEMP) pid_temp = HEATER_0_MINTEMP;
@@ -3309,7 +3309,9 @@ bool lcd_calibrate_z_end_stop_manual(bool only_z)
 
     // Until confirmed by the confirmation dialog.
     for (;;) {
-        const char   *msg                 = only_z ? _i("Calibrating Z. Rotate the knob to move the Z carriage up to the end stoppers. Click when done.") : _i("Calibrating XYZ. Rotate the knob to move the Z carriage up to the end stoppers. Click when done.");////MSG_MOVE_CARRIAGE_TO_THE_TOP c=20 r=8////MSG_MOVE_CARRIAGE_TO_THE_TOP_Z c=20 r=8
+        const char *msg = only_z
+            ? _i("Calibrating Z. Rotate the knob to move the Z carriage up to the end stoppers. Click when done.")////MSG_MOVE_CARRIAGE_TO_THE_TOP_Z c=20 r=8
+            : _i("Calibrating XYZ. Rotate the knob to move the Z carriage up to the end stoppers. Click when done.");////MSG_MOVE_CARRIAGE_TO_THE_TOP c=20 r=8
         const char   *msg_next            = lcd_display_message_fullscreen_P(msg);
         const bool    multi_screen        = msg_next != NULL;
         unsigned long previous_millis_msg = _millis();
@@ -4950,7 +4952,7 @@ void lcd_wizard(WizState state)
 				//current filament needs to be unloaded and then new filament should be loaded
 				//start to preheat nozzle for unloading remaining PLA filament
 				setTargetHotend(PLA_PREHEAT_HOTEND_TEMP, 0);
-				lcd_display_message_fullscreen_P(_i("Now I will preheat nozzle for PLA."));
+				lcd_display_message_fullscreen_P(_i("Now I will preheat nozzle for PLA.")); ////MSG_WIZARD_WILL_PREHEAT c=20 r=4
 				wait_preheat();
 				//unload current filament
 				unload_filament(true);
@@ -6662,7 +6664,7 @@ static void lcd_main_menu()
     }
     MENU_ITEM_SUBMENU_P(_i("Support"), lcd_support_menu);////MSG_SUPPORT c=18
 #ifdef LCD_TEST
-    MENU_ITEM_SUBMENU_P(_i("XFLASH init"), lcd_test_menu);////MSG_SUPPORT
+    MENU_ITEM_SUBMENU_P(_i("XFLASH init"), lcd_test_menu);////MSG_XFLASH
 #endif //LCD_TEST
 
     MENU_END();
@@ -6963,9 +6965,9 @@ static void lcd_control_temperature_menu()
 #if defined AUTOTEMP && (TEMP_SENSOR_0 != 0)
 //MENU_ITEM_EDIT removed, following code must be redesigned if AUTOTEMP enabled
   MENU_ITEM_EDIT(bool, MSG_AUTOTEMP, &autotemp_enabled);
-  MENU_ITEM_EDIT(float3, _i(" \002 Min"), &autotemp_min, 0, HEATER_0_MAXTEMP - 10);////MSG_MIN
+  MENU_ITEM_EDIT(float3, _i(" \xdf Min"), &autotemp_min, 0, HEATER_0_MAXTEMP - 10);////MSG_MIN
-  MENU_ITEM_EDIT(float3, _i(" \002 Max"), &autotemp_max, 0, HEATER_0_MAXTEMP - 10);////MSG_MAX
+  MENU_ITEM_EDIT(float3, _i(" \xdf Max"), &autotemp_max, 0, HEATER_0_MAXTEMP - 10);////MSG_MAX
-  MENU_ITEM_EDIT(float32, _i(" \002 Fact"), &autotemp_factor, 0.0, 1.0);////MSG_FACTOR
+  MENU_ITEM_EDIT(float32, _i(" \xdf Fact"), &autotemp_factor, 0.0, 1.0);////MSG_FACTOR
 #endif
 
   MENU_END();

PF-build.sh

@@ -1310,13 +1310,9 @@ create_multi_firmware()
             ./fw-clean.sh
             echo "$(tput sgr 0)"
         fi
-        # build languages
+        # Combine compiled firmware with languages
         echo "$(tput setaf 3)"
-        ./lang-build.sh || failures 25
-        # Combine compiled firmware with languages 
         ./fw-build.sh || failures 25
-        cp not_tran.txt not_tran_$VARIANT.txt
-        cp not_used.txt not_used_$VARIANT.txt
         echo "$(tput sgr 0)"
         # Check if the motherboard is an EINSY and if so only one hex file will generated
         MOTHERBOARD=$(grep --max-count=1 "\bMOTHERBOARD\b" $SCRIPT_PATH/Firmware/variants/$VARIANT.h | sed -e's/  */ /g' |cut -d ' ' -f3)
@@ -1324,7 +1320,7 @@ create_multi_firmware()
         if [ "$MOTHERBOARD" = "BOARD_EINSY_1_0a" ]; then
             echo "$(tput setaf 2)Copying multi language firmware for MK3/Einsy board to PF-build-hex folder$(tput sgr 0)"
             # End of "lang.bin" for MK3 and MK3S copy
-            cp -f firmware.hex $SCRIPT_PATH/../$OUTPUT_FOLDER/$OUTPUT_FILENAME.hex
+            cp -f Firmware-intl.hex $SCRIPT_PATH/../$OUTPUT_FOLDER/$OUTPUT_FILENAME.hex
             cp -f $BUILD_PATH/Firmware.ino.elf $SCRIPT_PATH/../$OUTPUT_FOLDER/$OUTPUT_FILENAME.elf
         else
             #Search for created firmware languages
@@ -1352,7 +1348,6 @@ create_multi_firmware()
     if [[ -z "$clean_flag" || "$clean_flag" == "0" ]]; then
         echo "$(tput setaf 3)"
         ./fw-clean.sh || failures 25
-        ./lang-clean.sh || failures 25
         echo "$(tput sgr 0)"
     fi
 }

build.sh

@@ -37,6 +37,4 @@ $BUILD_ENV_PATH/arduino $SCRIPT_PATH/Firmware/Firmware.ino --verify --board Prus
 export ARDUINO=$BUILD_ENV_PATH
 
 cd $SCRIPT_PATH/lang
-./lang-build.sh || exit 10
+./fw-build.sh || exit 10
-./lang-community.sh || exit 11
-./fw-build.sh || exit 12

lang/Add_new_language.md

@@ -1,194 +0,0 @@
-# How-to add a new language to Prusa Firmware
-
-We will use Dutch as an example here.
-
-## Prepare Prusa Firmware
-
-QR = placeholder for language in upper case
-
-qr = placeholder for language in lower case
-
-AB = placeholder for hexadecimal
-
-Files needs to be modified
-- `../Firmware/language.h` 
-
-  In section `/** @name Language codes (ISO639-1)*/` add the new `#define LANG_CODE_QR 0xABAB //!<'qr'`following ISO639-1 convention for QR.
-https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
-
-  Example:
-  `#define LANG_CODE_NL 0x6e6c //!<'nl'` where the hex value `0x6e6c` is in ascii `nl`
-
-- `../Firmware/language.c`
-
-  In section `const char* lang_get_name_by_code(uint16_t code)` add `case LANG_CODE_NL: return _n("Language");`
-
-  Example:
-
-  `case LANG_CODE_NL: return _n("Nederlands");` Where `Language` is native spoken version, here `Nederlands` (Netherlands) or `Vlaams` (Belgium). This will be displayed on the LCD menu.
-
-- `../lang/lang-add.sh`
-
-  In section `cat lang_add.txt | sed 's/^/"/;s/$/"/' | while read new_s; do` add `insert_qr "$new_s" 'qr'`where qr 
-  
-  Example:
-  `insert_qr "$new_s" 'nl'` with qr value `nl`for Dutch
-
-- `../lang/lang-build.sh`
-
-  In section `#returns hexadecimal data for lang code` add a case `*qr*) echo '0x71\0x72'`
-  
-  Example:
-  `*nl*) echo '\x6c\x6e' ;;` !!! IMPORTANT that the hex values are switched so 'nl' is here in 'ln' !!!
-  
-  In generate "all" section add `generate_binary 'qr'
-  
-  Example:
-  `generate_binary 'nl'`
-  
-- `../lang/lang-check.py`
-
-  Add in `help` the new language `qr`
-  
-  Example:
-  From `help="Check lang file (en|cs|de|es|fr|it|pl)")` to `help="Check lang file (en|cs|de|es|fr|nl|it|pl)")`
-  
-- In `../lang/lang-clean.sh`
-
-  In section echo `"lang-clean.sh started" >&2` add `clean_lang qr`
-  
-  Example:
-  `clean_lang nl`
-  
-
-- `../lang/lang-export.sh`
-
-  In section `# if 'all' is selected, script will generate all po files and also pot file` add `./lang-export.sh qr`
-  
-  Example:
-  
-  `./lang-export.sh nl`
-  
-  In section ` # language name in english` add `*qr*) echo "Language-in-English" ;;`
-  
-  Example:
-  `*nl*) echo "Dutch" ;;`
-  
-  
-
-  
-- `../lang/lang-import.sh`
-
-  In section `#replace in languages translation` add new rule set for the language.
-  As the LCD screen doesn't not support äöüßéè and other special characters, it makes sense to "normalize" these.
-  
-  Example:
-  ```
-  #replace in dutch translation according to https://nl.wikipedia.org/wiki/Accenttekens_in_de_Nederlandse_spelling
-  if [ "$LNG" = "nl" ]; then
-   #replace 'ë' with 'e'
-   sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
-   #replace 'ï' with 'i'
-   sed -i 's/\xc3\xaf/i/g' $LNG'_filtered.po'
-   #replace 'é' with 'e'
-   sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
-   #replace 'è' with 'e' (left)
-   sed -i 's/\xc3\xa8/e/g' $LNG'_filtered.po'
-   #replace 'ö' with 'o' (left)
-   sed -i 's/\xc3\xb6/o/g' $LNG'_filtered.po'
-   #replace 'ê' with 'e' (left)
-   sed -i 's/\xc3\xaa/e/g' $LNG'_filtered.po'
-   #replace 'ü' with 'u' (left)
-   sed -i 's/\xc3\xbc/u/g' $LNG'_filtered.po'
-   #replace 'ç' with 'c' (left)
-   sed -i 's/\xc3\xa7/c/g' $LNG'_filtered.po'
-   #replace 'á' with 'a' (left)
-   sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
-   #replace 'à' with 'a' (left)
-   sed -i 's/\xc3\xa0/a/g' $LNG'_filtered.po'
-   #replace 'ä' with 'a' (left)
-   sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
-   #replace 'û' with 'u' (left)
-   sed -i 's/\xc3\xbc/u/g' $LNG'_filtered.po'
-   #replace 'î' with 'i' (left)
-   sed -i 's/\xc3\xae/i/g' $LNG'_filtered.po'
-   #replace 'í' with 'i' (left)
-   sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
-   #replace 'ô' with 'o' (left)
-   sed -i 's/\xc3\xb4/o/g' $LNG'_filtered.po'
-   #replace 'ú' with 'u' (left)
-   sed -i 's/\xc3\xba/u/g' $LNG'_filtered.po'
-   #replace 'ñ' with 'n' (left)
-   sed -i 's/\xc3\xb1/n/g' $LNG'_filtered.po'
-   #replace 'â' with 'a' (left)
-   sed -i 's/\xc3\xa2/a/g' $LNG'_filtered.po'
-   #replace 'Å' with 'A' (left)
-   sed -i 's/\xc3\x85/A/g' $LNG'_filtered.po'
-  fi
-  ```
-
-- `../lang/fw-build.sh`
-
-  In section `#update _SEC_LANG in binary file if language is selected` add
-  ```
-  if [ -e lang_qr.bin ]; then
-   echo -n " Language-in-English  : " >&2
-   ./update_lang.sh qr 2>./update_lang_qr.out 1>/dev/null
-   if [ $? -eq 0 ]; then echo 'OK' >&2; else echo 'NG!' >&2; fi
-  fi
-  ```
- 
-  Example:
-  ```
-  if [ -e lang_nl.bin ]; then
-   echo -n " Dutch  : " >&2
-   ./update_lang.sh nl 2>./update_lang_nl.out 1>/dev/null
-  if [ $? -eq 0 ]; then echo 'OK' >&2; else echo 'NG!' >&2; fi
-  fi
-  ```
-  
-  In section `#create binary file with all languages` add `if [ -e lang_qr.bin ]; then cat lang_qr.bin >> lang.bin; fi`
-  
-  Example:
-  
-  `if [ -e lang_nl.bin ]; then cat lang_nl.bin >> lang.bin; fi`
-
-- `../lang/fw-clean.sh`
-
-  In section `echo "fw-clean.sh started" >&2` add
-  
-  ```
-  rm_if_exists firmware_qr.hex
-  ...
-  ...
-  rm_if_exists update_lang_qr.out
-  ```
-  
-  Example:
-  `rm_if_exists firmware_nl.hex`
-  
-  and
-  
-  `rm_if_exists update_lang_nl.out`
-
-
-## Prepare language part
-
-To prepare the actual language translation files we need create the `lang_en_qr.txt` file.
-1. Copy and `lang_en.txt` as `lang_en_qr.txt`
-2. run `../lang/lang-export.sh`
-3. copy `../lang/po/Firmware_qr.po` file to `../lang/po/new/qr.po`
-4. translate all messages using POEdit or other tools.
-5. use `lang/lang-import.sh qr` to generate `lang_en_qr.txt` from translated po files
-6. move `../lang/po/new/lang_en_qr.txt` to `../lang/lang_en_qr.txt`
-7. cleanup `../lang/po/new` folder by deleting
-   ```
-   qr_filtered.po
-   qr_new.po
-   nonascii.txt
-   ```
-
-
-
-
-##

lang/README.md

@@ -0,0 +1,96 @@
+# Internationalization support
+
+Multi-language support in the Prusa MK3 firmware is based on PO language files.
+
+Firmware support is controlled by the ``LANG_MODE`` define in the configuration, which defaults to 1 (enabled). When ``LANG_MODE`` is set, the firmware *can* load additional languages, but these extra languages need to be "baked in" in the firmware image for flashing. This last step is performed using the tools in this directory.
+
+## Quick reference
+
+### Required tools
+
+Python 3 with the ``regex``, ``pyelftools`` and ``polib`` modules as well as ``gettext`` and ``dos2unix``. On a debian-based distribution, install the required packages with:
+
+    sudo apt-get install python3-regex python3-pyelftools python3-polib gettext dos2unix
+
+### Main summary
+
+Language files:
+
+* ``po/Firmware.pot``: Main list of strings to translate (do *not* change this file manually - it is automatically generated)
+* ``po/Firmware_XY.po``: Translations for "XY", where XY is the ISO639-1 language code.
+
+PO files are simple text files that can be edited with any text editor, but are best handled using dedicated tools such as POEdit, Lokalize or Linguist.
+
+High-level tools:
+
+* ``config.sh``: Language selection/configuration
+* ``fw-build.sh``: Builds the final multi-language hex file into this directory
+* ``fw-clean.sh``: Cleanup temporary files left by ``fw-build.sh``
+* ``update-pot.sh``: Extract internationalized strings from the sources and place them inside ``po/Firmware.pot``
+* ``update-po.sh``: Refresh po file/s with new translations from the main pot file.
+
+Lower-level tools:
+
+* ``lang-check.py``: Checks a single po file for screen formatting issues.
+* ``lang-extract.py``: Extract internationalized strings from source files.
+* ``lang-map.py``: Extract and patch the translation symbol map in the firmware.
+* ``lang-build.py``: Build a binary language catalog for a single po file.
+* ``lang-patchsec.py``: Embed a single secondary language catalog in the firmware.
+
+### Building an internationalized firmware
+
+This is accomplished by running ``fw-build.sh`` after building the firmware with ``LANG_MODE = 1``. Language selection is done by modifying ``config.sh``.
+
+After running the script, the final ``Firmware-intl.hex`` will be generated in this directory.
+
+This step is already performed for you when using ``build.sh`` or ``PF-build.sh``. You can however re-run ``fw-build.sh`` to update just the language catalogs inside the image.
+
+### Updating an existing translation
+
+#### Typo or incorrect translation in existing text
+
+If you see a typo or an incorrect translation, simply edit ``po/Firmware_XY.po`` and make a pull request with the changes.
+
+You can use the following command:
+
+    ./lang-check.py po/Firmware_XY.po
+
+to check for screen formatting issues in isolation, or use the ``--information`` flag:
+
+    ./lang-check.py --information po/Firmware_XY.po
+
+to preview all translations as formatted on the screen.
+
+#### Missing translation without entry in po file
+
+If some text is missing, but there is no reference text in the po file, you need to refresh the translation file by picking up new strings and annotations from the template.
+
+Run ``./update-po.sh po/Firmware_XY.po`` to propagate the new strings to your language. This will merge the new strings, update references/annotations as well as marking unused strings as obsolete.
+
+Update the translations, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
+
+### Fixing an incorrect screen annotation or english text
+
+The screen annotations as well as the original english text is extracted from the firmware sources. **Do not change the main pot file**. The ``pot`` and ``po`` file contains the location of the annotation to help you fix the sources themselves.
+
+Run ``./update-pot.sh`` to regenerate ``po/Firmware.pot`` and verify that the annotation has been picked up correctly. You can stop here if you only care about the annotation.
+
+Run ``./update-po.sh po/Firmware_XY.po`` otherwise to propagate the annotation to your language, then proceed as for [typo or incorrect translation](#typo-or-incorrect-translation-in-existing-text).
+
+### Adding a new language
+
+Each language is assigned a two-letter ISO639-1 language code.
+
+The firmware needs to be aware of the language code. It's probably necessary to update the "Language codes" section in ``Firmware/language.h`` to add the new code as a ``LANG_CODE_XY`` define as well as add the proper language name in the function ``lang_get_name_by_code`` in ``Firmware/language.c``.
+
+It is a good idea to ensure the translation template is up-to-date before starting to translate. Run ``./update-pot.sh`` to regenerate ``po/Firmware.pot`` if possible.
+
+Copy ``po/Firmware.pot`` to ``po/Firmware_XY.po``. The *same* language code needs to be used for the "Language" entry in the metadata. Other entries can be customized freely.
+
+The new language needs to be explicitly added to the list of bundled languages in ``config.sh``.
+
+At this point the new language should be picked-up normally. See [how to build an internationalized firmware](#building-an-internationalized-firmware) and use ``lang-check.py`` for [previewing the translation](#typo-or-incorrect-translation-in-existing-text) without having to perform a complete rebuild.
+
+## Internal details
+
+TODO

lang/config.sh

@@ -34,42 +34,34 @@ if [ -z "$ARDUINO" ]; then
     export ARDUINO=../../PF-build-env-1.0.6/1.8.5-1.0.4-linux-64 #C:/arduino-1.8.5
 fi
 #
-# Arduino builder:
-if [ -z "$BUILDER" ]; then
-    export BUILDER=$ARDUINO/arduino-builder
-fi
-#
 # AVR gcc tools:
 if [ -z "$OBJCOPY" ]; then
     export OBJCOPY=$ARDUINO/hardware/tools/avr/bin/avr-objcopy
 fi
-if [ -z "$OBJDUMP" ]; then
-    export OBJDUMP=$ARDUINO/hardware/tools/avr/bin/avr-objdump
-fi
 #
 # Output folder:
 if [ -z "$OUTDIR" ]; then
     export OUTDIR="../../Prusa-Firmware-build"
 fi
 #
-# Objects folder:
+# Output elf file:
-if [ -z "$OBJDIR" ]; then
-    export OBJDIR="$OUTDIR/sketch"
-fi
-#
-# Generated elf file:
 if [ -z "$INOELF" ]; then
     export INOELF="$OUTDIR/Firmware.ino.elf"
 fi
 #
-# Generated hex file:
+# Output hex file:
 if [ -z "$INOHEX" ]; then
     export INOHEX="$OUTDIR/Firmware.ino.hex"
 fi
 #
+# Generated multi-language hex file prefix:
+if [ -z "$INTLHEX" ]; then
+    export INTLHEX="$LNGDIR/Firmware-intl"
+fi
+#
 # Set default languages
 if [ -z "$LANGUAGES" ]; then
-    export LANGUAGES="cz de es fr it pl"
+    export LANGUAGES="cs de es fr it pl"
 fi
 #
 # Check for community languages
@@ -87,48 +79,64 @@ elif [ "$COMMUNITY_LANG_GROUP" = "3" ]; then
     COMMUNITY_LANGUAGES=$(grep --max-count=$MAX_COMMINITY_LANG "^#define COMMUNITY_LANG_GROUP3_" $SRCDIR/Firmware/config.h| cut -d '_' -f4 |cut -d ' ' -f1 |tr '[:upper:]' '[:lower:]'| tr '\n' ' ')
 fi
 
+# End of customization
+######################
+
 if [ -z "$COMMUNITY_LANGUAGES" ]; then
     export COMMUNITY_LANGUAGES="$COMMUNITY_LANGUAGES"
 fi
 
-echo "$(tput setaf 2)config.sh started$(tput sgr0)" >&2
+if [ ! -t 2 -o "$TERM" = "dumb" ]; then
+    NO_COLOR=1
+fi
+
+color()
+{
+    color=$1
+    shift
+    if [ "$NO_COLOR" = 0 -o -z "$NO_COLOR" ]; then
+        echo "$(tput setaf $color)$*$(tput sgr 0)"
+    else
+        echo "$*"
+    fi
+}
+
+ok() { color 2 "OK"; }
+ng() { color 1 "NG!"; }
+
+color 2 "config.sh started" >&2
 
 _err=0
 
 echo -n " Arduino main folder: " >&2
-if [ -d $ARDUINO ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=1; fi
+if [ -d $ARDUINO ]; then ok >&2; else ng >&2; _err=1; fi
-
-echo -n " Arduino builder: " >&2
-if [ -e $BUILDER ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=2; fi
 
 echo " AVR gcc tools:" >&2
 echo -n "   objcopy " >&2
-if [ -e $OBJCOPY ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=3; fi
+if [ -e $OBJCOPY ]; then ok >&2; else ng >&2; _err=3; fi
-echo -n "   objdump " >&2
-if [ -e $OBJDUMP ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=4; fi
 
 echo -n " Output folder: " >&2
-if [ -d $OUTDIR ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=5; fi
+if [ -d $OUTDIR ]; then ok >&2; else ng >&2; _err=5; fi
 
-echo -n " Objects folder: " >&2
+echo -n " Output elf file: " >&2
-if [ -d $OBJDIR ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=6; fi
+if [ -e $INOELF ]; then ok >&2; else ng >&2; _err=7; fi
 
-echo -n " Generated elf file: " >&2
+echo -n " Output hex file: " >&2
-if [ -e $INOELF ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=7; fi
+if [ -e $INOHEX ]; then ok >&2; else ng >&2; _err=8; fi
 
-echo -n " Generated hex file: " >&2
+echo -n " Intl hex file prefix: " >&2
-if [ -e $INOHEX ]; then echo "$(tput setaf 2)OK$(tput sgr0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr0)" >&2; _err=8; fi
+if [ -n $INTLHEX ]; then ok >&2; else ng >&2; _err=8; fi
 
 echo -n " Languages: " >&2
-echo "$(tput setaf 2)$LANGUAGES$(tput sgr0)" >&2
+color 2 "$LANGUAGES" >&2
 
 echo -n " Community languages: " >&2
-echo "$(tput setaf 2)$COMMUNITY_LANGUAGES$(tput sgr0)" >&2
+color 2 "$COMMUNITY_LANGUAGES" >&2
 
 if [ $_err -eq 0 ]; then
- echo "$(tput setaf 2)config.sh finished with success$(tput sgr0)" >&2
+    color 2 "config.sh finished with success" >&2
- export CONFIG_OK=1
+    export CONFIG_OK=1
 else
- echo "$(tput setaf 1)config.sh finished with errors!$(tput sgr0)" >&2
+    color 1 "config.sh finished with errors!" >&2
- export CONFIG_OK=0
+    export CONFIG_OK=0
 fi

lang/fw-build.sh

@@ -1,209 +1,131 @@
 #!/bin/bash
-#
+# TODO: write some up-to-date description
-# Version 1.0.2 Build 12
+
-#
-# postbuild.sh - multi-language support script
-#  Generate binary with secondary language.
-#
-# Input files:
-#  $OUTDIR/Firmware.ino.elf
-#  $OUTDIR/sketch/*.o (all object files)
-#
-# Output files:
-#  text.sym
-#  $PROGMEM.sym (progmem1.sym)
-#  $PROGMEM.lss (...)
-#  $PROGMEM.hex
-#  $PROGMEM.chr
-#  $PROGMEM.var
-#  $PROGMEM.txt
-#  textaddr.txt
-#
-#############################################################################
-# Change log:
-# 31 May  2018, XPila,      Initial
-# 17 Dec. 2021, 3d-gussner, Use one config file for all languages
-# 11 Jan. 2022, 3d-gussner, Add check for not translated messages using a
-#                           parameter
-#                           Added version and Change log
-#                           colored output
-#                           Add Community language support
-#                           Use `git rev-list --count HEAD fw-build.sh`
-#                           to get Build Nr
-#############################################################################
-#
 # Config:
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
+if [ -z "$CONFIG_OK" ]; then source config.sh; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
+if [ -z "$CONFIG_OK" -o "$CONFIG_OK" -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
-#
-# Selected language:
-LNG=$1
-
-#Set default to ignore missing text
-CHECK_MISSING_TEXT=0
-#Check if script should check for missing messages in the source code aren't translated by using parameter "--check-missing-text"
-if [ "$1" = "--check-missing-text" ]; then
- CHECK_MISSING_TEXT=1
-fi
-
-# List of supported languages
-if [ -z "$LANGUAGES" ]; then
- LANGUAGES="cz de es fr it pl"
-fi
 
 # Community languages
 if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
-  LANGUAGES+=" $COMMUNITY_LANGUAGES"
+    LANGUAGES+=" $COMMUNITY_LANGUAGES"
 fi
-echo "$(tput setaf 2)fw-build.sh started$(tput sgr 0)" >&2
+
-echo "fw-build languages:$(tput setaf 2)$LANGUAGES$(tput sgr 0)" >&2
+color 2 "fw-build.sh started" >&2
+echo -n "fw-build languages: " >&2
+color 2 "$LANGUAGES" >&2
 
 finish()
 {
- echo
+    echo >&2
- if [ "$1" = "0" ]; then
+    if [ "$1" = "0" ]; then
-  echo "$(tput setaf 2)fw-build.sh finished with success$(tput sgr 0)" >&2
+        color 2 "fw-build.sh finished with success" >&2
- else
+    else
-  echo "$(tput setaf 1)fw-build.sh finished with errors!$(tput sgr 0)" >&2
+        color 1 "fw-build.sh finished with errors!" >&2
- fi
+    fi
- case "$-" in
+    case "$-" in
-  *i*) echo "press enter key"; read ;;
+    *i*)
- esac
+        echo "press enter key"; read ;;
- exit $1
+    esac
+    exit $1
 }
 
-#check input files
+# Clean the temporary directory
-echo " checking files:" >&2
+TMPDIR=$(dirname "$0")/tmp
-if [ ! -e $OUTDIR ]; then echo "$(tput setaf 1)  folder '$OUTDIR' not found!$(tput sgr 0)" >&2; finish 1; fi
+rm -rf "$TMPDIR"
-echo "$(tput setaf 2)  folder  OK$(tput sgr 0)" >&2
+mkdir -p "$TMPDIR"
-if [ ! -e $INOELF ]; then echo "$(tput setaf 1)  elf file '$INOELF' not found!$(tput sgr 0)" >&2; finish 1; fi
+BIN=$TMPDIR/firmware.bin
-echo "$(tput setaf 2)  elf     OK$(tput sgr 0)" >&2
+MAP=$TMPDIR/firmware.map
-if ! ls $OBJDIR/*.o >/dev/null 2>&1; then echo "$(tput setaf 1)  no object files in '$OBJDIR/'!$(tput sgr 0)" >&2; finish 1; fi
-echo "$(tput setaf 2)  objects OK$(tput sgr 0)" >&2
 
-#run progmem.sh - examine content of progmem1
+# Extract and patch the symbol table/language map
-echo -n " running progmem.sh..." >&2
+color 4 "generating firmware symbol map" >&2
-./progmem.sh 1 2>progmem.out
+"$OBJCOPY" -I ihex -O binary "$INOHEX" "$BIN"
-if [ $? -ne 0 ]; then echo "$(tput setaf 1)NG! - check progmem.out file$(tput sgr 0)" >&2; finish 1; fi
+./lang-map.py "$INOELF" "$BIN" > "$MAP"
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
 
-#run textaddr.sh - map progmem addreses to text identifiers
+# Get the maximum size of a single language table
-echo -n " running textaddr.sh..." >&2
+maxsize=$(grep '^#define \+LANG_SIZE_RESERVED \+' "$SRCDIR/Firmware/config.h" | sed -e 's/\s\+/ /g' | cut -d ' ' -f3)
-./textaddr.sh 2>textaddr.out
-if [ $? -ne 0 ]; then echo "$(tput setaf 1)NG! - check progmem.out file$(tput sgr 0)" >&2; finish 1; fi
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
 
-#check for messages declared in progmem1, but not found in lang_en.txt
+# Build language catalogs
-echo -n " checking textaddr.txt..." >&2
-cat textaddr.txt | grep "^TEXT NF" | sed "s/[^\"]*\"//;s/\"$//" >not_used.txt
-cat textaddr.txt | grep "^ADDR NF" | sed "s/[^\"]*\"//;s/\"$//" >not_tran.txt
-if cat textaddr.txt | grep "^ADDR NF" >/dev/null; then
- echo "$(tput setaf 1)NG! - some texts not found in lang_en.txt!$(tput sgr 0)"
- if [ $CHECK_MISSING_TEXT -eq 1 ]; then
-  echo "$(tput setaf 1)Missing text found, please update the language files!$(tput setaf 6)" >&2
-  cat not_tran.txt >&2
-  finish 1
- else
-  echo "$(tput setaf 3)  missing text ignored!$(tput sgr 0)" >&2
- fi
-else
- echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
-fi
-
-#extract binary file
-echo -n " extracting binary..." >&2
-$OBJCOPY -I ihex -O binary $INOHEX ./firmware.bin
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
-
-#update binary file
-echo " updating binary:" >&2
-
-#update progmem1 id entries in binary file
-echo -n "  primary language ids..." >&2
-cat textaddr.txt | grep "^ADDR OK" | cut -f3- -d' ' | sed "s/^0000/0x/" |\
- awk '{ id = $2 - 1; hi = int(id / 256); lo = int(id - 256 * hi); printf("%d \\\\x%02x\\\\x%02x\n", strtonum($1), lo, hi); }' |\
- while read addr data; do
-  /bin/echo -n -e $data | dd of=./firmware.bin bs=1 count=2 seek=$addr conv=notrunc oflag=nonblock 2>/dev/null
- done
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
-
-#update primary language signature in binary file
-echo -n "  primary language signature..." >&2
-if [ -e lang_en.bin ]; then
- #find symbol _PRI_LANG_SIGNATURE in section '.text'
- pri_lang=$(cat text.sym | grep -E "\b_PRI_LANG_SIGNATURE\b")
- if [ -z "$pri_lang" ]; then echo "$(tput setaf 1)NG!\n  symbol _PRI_LANG_SIGNATURE not found!$(tput sgr 0)" >&2; finish 1; fi
- #get pri_lang address
- pri_lang_addr='0x'$(echo $pri_lang | cut -f1 -d' ')
- #read header from primary language binary file
- header=$(dd if=lang_en.bin bs=1 count=16 2>/dev/null | xxd | cut -c11-49 | sed 's/\([0-9a-f][0-9a-f]\)[\ ]*/\1 /g')
- #read checksum and count data as 4 byte signature
- chscnt=$(echo $header | cut -c18-29 | sed "s/ /\\\\x/g")
- /bin/echo -e -n "$chscnt" |\
-  dd of=firmware.bin bs=1 count=4 seek=$(($pri_lang_addr)) conv=notrunc 2>/dev/null
- echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
-else
- echo "$(tput setaf 1)NG! - file lang_en.bin not found!$(tput sgr 0)" >&2;
- finish 1
-fi
-
-#convert bin to hex
-echo -n " converting primary to hex..." >&2
-$OBJCOPY -I binary -O ihex ./firmware.bin ./firmware.hex
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
-
-#update _SEC_LANG in binary file if language is selected
-echo -n " secondary language data..." >&2
-if [ ! -z "$LNG" ]; then
- ./update_lang.sh $LNG 2>./update_lang.out
- if [ $? -ne 0 ]; then echo "$(tput setaf 1)NG! - check update_lang.out file$(tput sgr 0)" >&2; finish 1; fi
- echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
- finish 0
-else
- echo >&2
- echo "  Updating languages:" >&2
- for lang in $LANGUAGES; do
-  if [ -e lang_$lang.bin ]; then
-   echo -n "   $lang  : " >&2
-   ./update_lang.sh $lang 2>./update_lang_$lang.out 1>/dev/null
-   if [ $? -eq 0 ]; then echo "$(tput setaf 2)OK$(tput sgr 0)" >&2; else echo "$(tput setaf 1)NG!$(tput sgr 0)" >&2; finish 1; fi
-  fi
- done 
-fi
-
-#create binary file with all languages
-rm -f lang.bin
 for lang in $LANGUAGES; do
- if [ -e lang_$lang.bin ]; then cat lang_$lang.bin >> lang.bin; fi
+    pofile="po/Firmware_$lang.po"
+    binfile="$TMPDIR/lang_$lang.bin"
+
+    color 4 "compiling language \"$lang\" from $pofile" >&2
+    ./lang-check.py --map "$MAP" "$pofile"
+    if [ "$?" != 0 ]; then
+        color 1 "$pofile: NG! - translation contains warnings or errors" >&2
+    fi
+
+    ./lang-build.py "$MAP" "$pofile" "$binfile"
+
+    # ensure each catalog fits the reserved size
+    if [[ $(stat -c '%s' "$binfile") -gt $maxsize ]]; then
+        color 1 "$pofile: NG! - language data exceeds $maxsize bytes" >&2
+        finish 1
+    fi
 done
 
-# Check that the language data doesn't exceed the reserved XFLASH space
+# Detect the printer type and choose the language type
-echo " checking language data size:"
+if grep -q '^#define \+PRINTER_TYPE \+PRINTER_\(MK25\|MK25S\)\b' "$SRCDIR/Firmware/Configuration_prusa.h"; then
-lang_size=$(wc -c lang.bin | cut -f1 -d' ')
+    has_xflash=0
-lang_size_pad=$(( ($lang_size+4096-1) / 4096 * 4096 ))
-
-# TODO: hard-coded! get value by preprocessing LANG_SIZE from xflash_layout.h!
-lang_reserved=249856
-
-echo -n "  total size usage: " >&2
-if [ $lang_size_pad -gt $lang_reserved ]; then
-  echo -n "$(tput setaf 1)" >&2
 else
-  echo -n "$(tput setaf 2)" >&2
+    has_xflash=1
-fi
-echo "$lang_size_pad ($lang_size)$(tput sgr 0)" >&2
-echo "  reserved size:    $(tput setaf 2)$lang_reserved$(tput sgr 0)" >&2
-if [ $lang_size_pad -gt $lang_reserved ]; then
-  echo "$(tput setaf 1)NG! - language data too large$(tput sgr 0)" >&2
-  finish 1
 fi
 
-#convert lang.bin to lang.hex
+if [ "$has_xflash" = 1 ]; then
-echo -n " converting multi language to hex..." >&2
+    # Build the final hex file with XFLASH support (catalogs appended to a single hex file)
-$OBJCOPY -I binary -O ihex ./lang.bin ./lang.hex
+    OUTHEX="${INTLHEX}.hex"
-echo "$(tput setaf 2)OK$(tput sgr 0)" >&2
 
-#append languages to hex file
+    color 4 "assembling final firmware image" >&2
-cat ./lang.hex >> firmware.hex
+    "$OBJCOPY" -I binary -O ihex "$BIN" "$OUTHEX"
+    truncate -s0 "$TMPDIR/lang.bin"
+    for lang in $LANGUAGES; do
+        cat "$TMPDIR/lang_$lang.bin" >> "$TMPDIR/lang.bin"
+    done
+    "$OBJCOPY" -I binary -O ihex "$TMPDIR/lang.bin" "$TMPDIR/lang.hex"
+    cat "$TMPDIR/lang.hex" >> "$OUTHEX"
+
+    # Check that the language data doesn't exceed the reserved XFLASH space
+    lang_size=$(stat -c '%s' "$TMPDIR/lang.bin")
+    lang_size_pad=$(( ($lang_size+4096-1) / 4096 * 4096 ))
+
+    # TODO: hard-coded! get value by preprocessing LANG_SIZE from xflash_layout.h!
+    lang_reserved=249856
+
+    echo >&2
+    echo -n "  total size usage: " >&2
+    [ $lang_size_pad -gt $lang_reserved ] && c=1 || c=2
+    color $c "$lang_size_pad ($lang_size)" >&2
+    echo -n "     reserved size: " >&2
+    color 2 "$lang_reserved" >&2
+    if [ $lang_size_pad -gt $lang_reserved ]; then
+        color 1 "NG! - language data too large" >&2
+        finish 1
+    fi
+
+    echo -n "  multilanguage fw: " >&2
+    color 2 "$OUTHEX" >&2
+else
+    # Build one hex file for each secondary language
+    color 4 "assembling final firmware images" >&2
+    echo >&2
+    echo -n "  maximum size: " >&2
+    color 2 "$(( $maxsize ))" >&2
+
+    for lang in $LANGUAGES; do
+        OUTHEX="${INTLHEX}-en_${lang}.hex"
+        catfile="$TMPDIR/lang_$lang.bin"
+        bintmp="$TMPDIR/fw-en_$lang.bin"
+
+        # patch the secondary language table
+        cp "$BIN" "$bintmp"
+        ./lang-patchsec.py "$INOELF" "$catfile" "$bintmp"
+        "$OBJCOPY" -I binary -O ihex "$bintmp" "$OUTHEX"
+
+        # print some stats
+        catsize=$(stat -c '%s' "$catfile")
+        echo -n "  $lang: " >&2
+        color 2 "$(printf "%5d %s" "$catsize" "$OUTHEX")" >&2
+    done
+fi
 
 finish 0

lang/fw-clean.sh

@@ -1,84 +1,13 @@
 #!/bin/bash
-#
+set -e
-# Version 1.0.1 Build 11
-#
-# fw-clean.sh - multi-language support script
-#  Remove all firmware output files from lang folder.
-#
-#############################################################################
-# Change log:
-# 21 June 2018, XPila,      Initial
-# 11 Sep. 2018, XPila,      Lang update, french translation
-#                           resized reserved space
-# 18 Oct. 2018, XPila,      New lang, arduino 1.8.5 - fw-clean.sh and lang-clean.sh fix
-# 10 Dec. 2018, jhoblitt,   make all shell scripts executable
-# 26 Jul. 2019, leptun,     Fix shifted languages. Use \n and \x0a
-# 14 Sep. 2019, 3d-gussner, Prepare adding new language
-# 01 Mar. 2021, 3d-gussner, Move `Dutch` language parts
-# 22 Mar. 2021, 3d-gussner, Move Dutch removing part to correct loaction
-# 17 Dec. 2021, 3d-gussner, Use one config file for all languages
-# 11 Jan. 2022, 3d-gussner, Added version and Change log
-#                           colored output
-#                           Use `git rev-list --count HEAD fw-clean.sh`
-#                           to get Build Nr
-# 25 Jan. 2022, 3d-gussner, Update documentation
-#############################################################################
-# Config:
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
 
-if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
+# Config
-  LANGUAGES+=" $COMMUNITY_LANGUAGES"
+if [ -z "$CONFIG_OK" ]; then source config.sh; fi
-fi
+if [ -z "$CONFIG_OK" -o "$CONFIG_OK" -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
-echo "$(tput setaf 2)fw-clean.sh started$(tput sgr0)" >&2
-echo "fw-clean languages:$(tput setaf 2)$LANGUAGES$(tput sgr0)" >&2
 
-result=0
+# Clean the temporary directory
+TMPDIR=$(dirname "$0")/tmp
+rm -rf "$TMPDIR"
 
-rm_if_exists()
+# Remove internationalized firmware files
-{
+rm -f "${INTLHEX}"*.hex
- if [ -e $1 ]; then
-  echo -n "$(tput sgr0) removing $(tput sgr0) '$1'..." >&2
-  if rm $1; then
-   echo "$(tput setaf 2)OK$(tput sgr0)" >&2
-  else
-   echo "$(tput setaf 1)NG!$(tput sgr0)" >&2
-   result=1
-  fi
- fi
-}
-
-
-rm_if_exists text.sym
-rm_if_exists progmem1.sym
-rm_if_exists progmem1.lss
-rm_if_exists progmem1.hex
-rm_if_exists progmem1.chr
-rm_if_exists progmem1.var
-rm_if_exists progmem1.txt
-rm_if_exists textaddr.txt
-rm_if_exists firmware.bin
-rm_if_exists firmware.hex
-rm_if_exists progmem.out
-rm_if_exists textaddr.out
-rm_if_exists update_lang.out
-rm_if_exists lang.bin
-rm_if_exists lang.hex
-
-
-for lang in $LANGUAGES; do
- rm_if_exists firmware_$lang.hex
- rm_if_exists update_lang_$lang.out
-done
-
-if [ $result -eq 0 ]; then
- echo "$(tput setaf 2)fw-clean.sh finished with success$(tput sgr0)" >&2
-else
- echo "$(tput setaf 1)fw-clean.sh finished with errors!$(tput sgr0)" >&2
-fi
-
-case "$-" in
- *i*) echo "press enter key"; read ;;
-esac
-
-exit $result

lang/lang-add.sh

@@ -1,99 +0,0 @@
-#!/bin/bash
-#
-# lang-add.sh - multi-language support script
-#  add new texts from list (lang_add.txt) to all dictionary files
-#
-# Input files:
-#  lang_add.txt
-# Updated files:
-#  lang_en.txt and all lang_en_xx.txt
-
-# List of supported languages
-LANGUAGES="cz de es fr it pl"
-
-# Community languages
-LANGUAGES+=" nl" #Dutch
-
-# Config:
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo 'Config NG!' >&2; exit 1; fi
-
-if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
-  LANGUAGES+=" $COMMUNITY_LANGUAGES"
-fi
-LANGUAGES=$(ls lang_en_*.txt|cut -d '_' -f3|cut -d '.' -f1)
-echo "lang-add languages:$LANGUAGES" >&2
-
-# insert single text to english dictionary
-# $1 - text to insert
-# $2 - metadata
-insert_en()
-{
-	#replace '[' and ']' in string with '\[' and '\]'
-	str=$(echo "$1" | sed 's/\[/\\\[/g;s/\]/\\\]/g')
-	# extract english texts, merge new text, grep line number
-	ln=$((cat lang_en.txt; echo "$1") | sed "/^$/d;/^#/d" | sort | grep -n "$str" | sed "s/:.*//;q")
-	# calculate position for insertion
-	ln=$((3*(ln-2)+1))
-	[ "$ln" -lt 1 ] && ln=1
-	# insert new text
-	sed -i "$ln"'i\\' lang_en.txt
-	sed -i "$ln"'i\'"$1"'\' lang_en.txt
-	sed -i "${ln}i\\#$2" lang_en.txt
-}
-
-# insert single text to translated dictionary
-# $1 - text to insert
-# $2 - suffix
-# $3 - metadata
-insert_xx()
-{
-	#replace '[' and ']' in string with '\[' and '\]'
-	str=$(echo "$1" | sed 's/\[/\\\[/g;s/\]/\\\]/g')
-	# extract english texts, merge new text, grep line number
-	ln=$((cat lang_en_$2.txt; echo "$1") | sed "/^$/d;/^#/d" | sed -n 'p;n' | sort | grep -n "$str" | sed "s/:.*//;q")
-	# calculate position for insertion
-	ln=$((4*(ln-2)+1))
-	[ "$ln" -lt 1 ] && ln=1
-	# insert new text
-	sed -i "$ln"'i\\' lang_en_$2.txt
-	sed -i "$ln"'i\"\x00"\' lang_en_$2.txt
-	sed -i "$ln"'i\'"$1"'\' lang_en_$2.txt
-	sed -i "${ln}i\\#$3" lang_en_$2.txt
-}
-
-# find the metadata for the specified string
-# TODO: this is unbeliveably crude
-# $1 - text to search for
-find_metadata()
-{
-    FIND_STR=$(echo $1|sed 's/\\/\\\\/g;s/\\\\x0a/\\\\n/g')
-    sed -ne "s^.*\(_[iI]\|ISTR\)($FIND_STR).*////\(.*\)^\2^p" ../Firmware/*.[ch]* | head -1
-}
-
-# check if input file exists
-if ! [ -e lang_add.txt ]; then
-	echo "file lang_add.txt not found"
-	exit 1
-fi
-
-cat lang_add.txt | sed 's/^/"/;s/$/"/;s/\\/\\\\/g' | while read new_s; do
-	if grep "$new_s" lang_en.txt >/dev/null; then
-		echo "text already exist:"
-		echo "$new_s"
-		echo
-	else
-		meta=$(find_metadata "$new_s")
-		echo "adding text:"
-		echo "$new_s ($meta)"
-		echo
-
-		insert_en "$new_s" "$meta"
-        for lang in $LANGUAGES; do
-            insert_xx "$new_s" "$lang" "$meta"
-        done
-	fi
-done
-
-read -t 5
-exit 0

lang/lang-build.py

@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+from collections import defaultdict
+import codecs
+import argparse
+import os
+import polib
+import struct
+import sys
+
+import lib.charset as cs
+from lib.io import info, warn, fatal, load_map
+
+FW_MAGIC = 0x4bb45aa5
+
+
+def translation_ref(translation):
+    cmt = translation.comment
+    if cmt and cmt.startswith('MSG_'):
+        return cmt.split(' ', 1)[0]
+    else:
+        return repr(translation.msgid)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--warn-unused', action='store_true',
+                    help='Warn about unused translations')
+    ap.add_argument('--show-coalesced', action='store_true',
+                    help='List coalesced translations')
+    ap.add_argument('map', help='Firmware symbol map file')
+    ap.add_argument('po', help='PO file')
+    ap.add_argument('out', help='output')
+    args = ap.parse_args()
+
+    # check arguments
+    for path in [args.map, args.po]:
+        if not os.path.isfile(path):
+            fatal("{} does not exist or is not a regular file".format(args.po))
+
+    # load the map file
+    syms = load_map(args.map)
+    fw_sig_data = None
+    msgid_data = defaultdict(list)
+    id_msgid = {}
+    sym_cnt = 0
+    for sym in syms:
+        if sym['name'] == '_PRI_LANG_SIGNATURE':
+            fw_sig_data = sym['data']
+        else:
+            # redo forward text transformation for transparent matching
+            msgid = cs.source_to_unicode(codecs.decode(sym['data'], 'unicode_escape', 'strict'))
+            msgid_data[msgid].append(sym)
+            id_msgid[sym['id']] = msgid
+
+            # update the max symbol count
+            if sym_cnt <= sym['id']:
+                sym_cnt = sym['id'] + 1
+
+    if fw_sig_data is None:
+        fatal('_PRI_LANG_SIGNATURE not found in map')
+
+    # open translations
+    po = polib.pofile(args.po)
+    lang_code = po.metadata['Language']
+    if not lang_code.isascii() or len(lang_code) != 2:
+        fatal(f'unsupported language code {lang_code}')
+
+    # build a catalog of all translations
+    trans_table = {}
+    for translation in po:
+        msgid = translation.msgid
+        found = msgid in msgid_data
+        if found:
+            trans_table[msgid] = (translation, msgid_data[msgid])
+        elif args.warn_unused:
+            err = "{}:{}".format(args.po, translation.linenum)
+            err += ": unused translation "
+            err += translation_ref(translation)
+            warn(err)
+
+    for msgid, syms in msgid_data.items():
+        if msgid not in trans_table:
+            # warn about missing translations
+            warn("untranslated text: " + repr(msgid))
+
+    # write the binary catalog
+    with open(args.out, "w+b") as fd:
+        fixed_offset = 16+2*sym_cnt
+        written_locs = {}
+
+        # compute final data tables
+        offsets = b''
+        strings = b'\0'
+        for i in range(sym_cnt):
+            msgid = id_msgid.get(i)
+            translation = trans_table.get(msgid)
+            if translation is None or len(translation[0].msgstr) == 0 or translation[0].msgstr == msgid:
+                # first slot reserved for untraslated/identical entries
+                offsets += struct.pack("<H", fixed_offset)
+            else:
+                string_bin = cs.unicode_to_source(translation[0].msgstr)
+
+                # check for invalid characters
+                invalid_char = cs.translation_check(string_bin)
+                if invalid_char is not None:
+                    line = translation[0].linenum
+                    warn(f'{args.po}:{line} contains unhandled character ' + repr(invalid_char))
+
+                string_bin = string_bin.encode('raw_unicode_escape', 'ignore')
+                string_off = written_locs.get(string_bin)
+                offset = fixed_offset + len(strings)
+                if string_off is not None:
+                    # coalesce repeated strings
+                    if args.show_coalesced:
+                        info(f'coalescing {offset:04x}:{string_off:04x} {string_bin}')
+                    offset = string_off
+                else:
+                    # allocate a new string
+                    written_locs[string_bin] = offset
+                    strings += string_bin + b'\0'
+                offsets += struct.pack("<H", offset)
+
+        # header
+        size = 16 + len(offsets) + len(strings)
+        header = struct.pack(
+            "<IHHHHI",
+            FW_MAGIC,
+            size,
+            sym_cnt,
+            0, # no checksum yet
+            (ord(lang_code[0]) << 8) + ord(lang_code[1]),
+            fw_sig_data)
+
+        fd.write(header)
+        fd.write(offsets)
+        fd.write(strings)
+
+        # calculate and update the checksum
+        cksum = 0
+        fd.seek(0)
+        for i in range(size):
+            cksum += (ord(fd.read(1)) << (0 if i % 2 else 8))
+            cksum &= 0xffff
+        fd.seek(8)
+        fd.write(struct.pack("<H", cksum))
+
+    return 0
+
+
+if __name__ == '__main__':
+    exit(main())

lang/lang-build.sh

@@ -1,247 +0,0 @@
-#!/bin/bash
-#
-# Version 1.0.2 Build 28
-#
-# lang-build.sh - multi-language support script
-#  generate lang_xx.bin (language binary file)
-#
-# Input files:
-#  lang_en.txt or lang_en_xx.txt
-#
-# Output files:
-#  lang_xx.bin
-#
-# Depending on files:
-#  ../Firmware/config.h to read the max allowed size for translations
-#
-# Temporary files:
-#  lang_en.cnt //calculated number of messages in english
-#  lang_en.max //maximum size determined by reading "../Firmware/config.h"
-#  lang_xx.tmp
-#  lang_xx.dat
-#
-#############################################################################
-# Change log:
-# 18 June 2018, XPila,      Initial
-# 19 June 2018, XPila,      New ML support
-# 18 Oct. 2018, XPila,      New lang French
-# 26 Nov. 2018, mkbel,      Automate secondary language support build.
-#  7 May  2019, ondratu     Check translation dictionary files to display definition
-# 19 June 2019, mkbel       Disable language check warnings of type "[W]: No display definition on line".
-#                           Those warnings were masking all other much more useful build process output.
-# 14 Sep. 2019, 3d-gussner, Prepare adding new language
-# 18 Sep. 2020, 3d-gussner, Update new messages and their translations, fix translations
-#                           Update CZ, FR, IT, ES translations
-#                             CZ thanks to @DRracer
-#                             FR thanks to Carlin Dcustom
-#                             ES
-#                             IT thanks to @wavexx
-#                           Co-authored-by: @DRracer, @wavexx
-#  1 Mar. 2021, 3d-gussner, Add Dutch translation
-# 17 Dec. 2021, 3d-gussner, Use one config file for all languages
-# 21 Dec. 2021, 3d-gussner, Prepare more community languages
-#                             Swedish
-#                             Danish
-#                             Slovanian
-#                             Hungarian
-#                             Luxembourgian
-#                             Croatian
-#  3 Jan. 2022, 3d-gussner, Prepare Lithuanian
-#                           Cleanup outdated code
-# 11 Jan. 2022, 3d-gussner, Add message and size count comparison
-#                           Added version and Change log
-#                           colored output
-#                           Add Community language support
-#                           Use `git rev-list --count HEAD lang-build.sh`
-#                           to get Build Nr
-# 25 Jan. 2022, 3d-gussner, Fix check
-#                           Update documentation
-# 10 Feb. 2022, 3d-gussner, Use SRCDIR for compatibility with build server
-# 11 Feb. 2022, 3d-gussner, Change to python3
-#############################################################################
-#
-# Config:
-
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr 0)" >&2; exit 1; fi
-
-if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
-  LANGUAGES+=" $COMMUNITY_LANGUAGES"
-fi
-
-#startup message
-echo "$(tput setaf 2)lang-build.sh started$(tput sgr 0)" >&2
-echo "lang-build languages:$(tput setaf 2)$LANGUAGES$(tput sgr 0)" >&2
-
-#awk code to format ui16 variables for dd
-awk_ui16='{ h=int($1/256); printf("\\x%02x\\x%02x\n", int($1-256*h), h); }'
-
-
-#exiting function
-finish()
-{
- if [ $1 -eq 0 ]; then
-  echo "$(tput setaf 2)lang-build.sh finished with success$(tput sgr 0)" >&2
- else
-  echo "$(tput setaf 1)lang-build.sh finished with errors!$(tput sgr 0)" >&2
- fi
- exit $1
-}
-
-#returns hexadecial data for lang code
-lang_code_hex_data()
-# $1 - language code ('en', 'cz'...)
-{
- case "$1" in
-  *en*) echo '\x6e\x65' ;;
-  *cz*) echo '\x73\x63' ;;
-  *de*) echo '\x65\x64' ;;
-  *es*) echo '\x73\x65' ;;
-  *fr*) echo '\x72\x66' ;;
-  *it*) echo '\x74\x69' ;;
-  *pl*) echo '\x6c\x70' ;;
-#Community language support
-#Dutch
-  *nl*) echo '\x6c\x6e' ;;
-#Swedish
-  *sv*) echo '\x76\x73' ;;
-#Norwegian
-  *no*) echo '\x6f\x6e' ;;
-#Danish
-  *da*) echo '\x61\x64' ;;
-#Slovak
-  *sk*) echo '\x6b\x73' ;;
-#Slovanian
-  *sl*) echo '\x6c\x73' ;;
-#Hungarian
-  *hu*) echo '\x75\x68' ;;
-#Luxembourgish
-  *lb*) echo '\x62\x6c' ;;
-#Croatian
-  *hr*) echo '\x72\x68' ;;
-#Lithuanian
-  *lt*) echo '\x74\x6c' ;;
-#Romanian
-  *ro*) echo '\x6f\x72' ;;
-#Use the 2 lines below as a template and replace 'qr' and `\x71\x72`
-##New language
-#  *qr*) echo '\x71\x72' ;;
- esac
- echo '??'
-}
-
-write_header()
-# $1 - lang
-# $2 - size
-# $3 - count
-# $4 - checksum
-# $5 - signature
-{
- /bin/echo -n -e "\xa5\x5a\xb4\x4b" |\
-  dd of=lang_$1.bin bs=1 count=4 seek=0 conv=notrunc 2>/dev/null
- /bin/echo -n -e $(echo -n "$(($2))" | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=4 conv=notrunc 2>/dev/null
- /bin/echo -n -e $(echo -n "$(($3))" | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=6 conv=notrunc 2>/dev/null
- /bin/echo -n -e $(echo -n "$(($4))" | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=8 conv=notrunc 2>/dev/null
- /bin/echo -n -e "$(lang_code_hex_data $1)" |\
-  dd of=lang_$1.bin bs=1 count=2 seek=10 conv=notrunc 2>/dev/null
- sig_h=$(($5 / 65536))
- /bin/echo -n -e $(echo -n "$sig_h" | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=14 conv=notrunc 2>/dev/null
- sig_l=$(($5 - $sig_h * 65536))
- /bin/echo -n -e $(echo -n "$sig_l" | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=12 conv=notrunc 2>/dev/null
-}
-
-generate_binary()
-# $1 - language code ('en', 'cz'...)
-{
- echo "lang=$(tput setaf 2)$1$(tput sgr 0)" >&2
- #remove output and temporary files
- rm -f lang_$1.bin
- rm -f lang_$1.tmp
- rm -f lang_$1.dat
- LNG=$1
- #check lang dictionary
- python3 lang-check.py $1 #--no-warning
- #create lang_xx.tmp - different processing for 'en' language
- if [[ "$1" = "en" || ! -f "lang_en.max" ]]; then
-  #remove comments and empty lines
-  cat lang_en.txt | sed '/^$/d;/^#/d'
-  #calculate number of strings
-  count=$(grep -c '^"' lang_en.txt)
-  echo "count="$count >&2
-  #Calculate the number of strings and save to temporary file
-  echo $count >lang_en.cnt
-  #read the allowed maxsize from "../Firmware/config.h" and save to temporary file
-  maxsize=$(($(grep "#define LANG_SIZE_RESERVED" $SRCDIR/Firmware/config.h|sed -e's/  */ /g' |cut -d ' ' -f3)))
-
-  echo "maxsize="$maxsize >&2
-  echo $maxsize >lang_en.max
- else
-  #remove comments and empty lines, print lines with translated text only
-  cat lang_en_$1.txt | sed '/^$/d;/^#/d' | sed -n 'n;p'
- fi | sed 's/^\"\\x00\"$/\"\"/' > lang_$1.tmp
- #create lang_xx.dat (binary text data file)
-# cat lang_$1.tmp | sed 's/^\"/\/bin\/echo -e \"/;s/"$/\\x00\"/' > lang_$1.shx
- cat lang_$1.tmp | sed 's/^\"/\/bin\/echo -e -n \"/;s/"$/\\x00\"/' | sh >lang_$1.dat
- #calculate number of strings
- count=$(grep -c '^"' lang_$1.tmp)
- echo "count="$count >&2
- # read string count of English and compare it with the translation
- encount=$(cat lang_en.cnt)
- if [ "$count" -eq "$encount" ]; then
-	echo "$(tput setaf 2)OK:"$1"="$count"$(tput sgr 0) is equal to $(tput setaf 2)en="$encount"$(tput sgr 0)" >&2
- else
-	echo "$(tput setaf 1)Error:"$1"="$count"$(tput sgr 0) is NOT equal to $(tput setaf 1)en="$encount"$(tput sgr 0)" >&2
-	finish 1
- fi
- #calculate text data offset
- offs=$((16 + 2 * $count))
- echo "offs="$offs >&2
- #calculate text data size
- size=$(($offs + $(wc -c lang_$1.dat | cut -f1 -d' ')))
- echo "size="$size >&2
- # read maxsize and compare with the translation
- maxsize=$(cat lang_en.max)
- if [ "$size" -lt "$maxsize" ]; then
-	free_space=$(($maxsize - $size))
-	echo "$(tput setaf 2)OK:"$1"="$size"$(tput sgr 0) is less than $(tput setaf 2)"$maxsize"$(tput sgr 0). Free space:$(tput setaf 2)"$free_space"$(tput sgr 0)" >&2
- else
-	echo "$(tput setaf 1)Error:"$1"="$size"$(tput sgr 0) is higer than $(tput setaf 3)"$maxsize"$(tput sgr 0)" >&2
-	finish 1
- fi
- #write header with empty signature and checksum
- write_header $1 $size $count 0x0000 0x00000000
- #write offset table
- offs_hex=$(cat lang_$1.tmp | sed 's/^\"//;s/\"$//' |\
-  sed 's/\\x[0-9a-f][0-9a-f]/\./g;s/\\[0-7][0-7][0-7]/\./g;s/\ /\./g' |\
-  awk 'BEGIN { o='$offs';} { h=int(o/256); printf("\\x%02x\\x%02x",int(o-256*h), h); o+=(length($0)+1); }')
- /bin/echo -n -e "$offs_hex" | dd of=./lang_$1.bin bs=1 seek=16 conv=notrunc 2>/dev/null
- #write binary text data
- dd if=./lang_$1.dat of=./lang_$1.bin bs=1 seek=$offs conv=notrunc 2>/dev/null
- #write signature
- if [ "$1" != "en" ]; then
-  dd if=lang_en.bin of=lang_$1.bin bs=1 count=4 skip=6 seek=12 conv=notrunc 2>/dev/null
- fi
- #calculate and update checksum
- chsum=$(cat lang_$1.bin | xxd | cut -c11-49 | tr ' ' "\n" | sed '/^$/d' | awk 'BEGIN { sum = 0; } { sum += strtonum("0x"$1); if (sum > 0xffff) sum -= 0x10000; } END { printf("%x\n", sum); }')
- /bin/echo -n -e $(echo -n $((0x$chsum)) | awk "$awk_ui16") |\
-  dd of=lang_$1.bin bs=1 count=2 seek=8 conv=notrunc 2>/dev/null
-}
-
-if [ -z "$1" ]; then set 'all'; fi
-
-if [ "$1" = "all" ]; then
- generate_binary 'en'
- for lang in $LANGUAGES; do
-  echo " Running : $lang" >&2
-  generate_binary $lang
- done
-else
- generate_binary $1
-fi
-
-finish 0

lang/lang-check.py

@@ -28,28 +28,23 @@
 #                                               newly import `lang_en_??.txt` files
 # 14 Mar. 2022, 3d-gussner, Check if translation isn't equal to origin
 #############################################################################
-#
+
-# Expected syntax of the files, which other scripts depend on
+"""Check PO files for formatting errors."""
-# 'lang_en.txt'
-# 1st line: '#MSG_'<some text>' c='<max chars in a column>' r='<max rows> ; '#MSG' is mandentory while 'c=' and 'r=' aren't but should be there
-# 2nd line: '"'<origin message used in the source code>'"' ; '"' double quotes at the beginning and end of message are mandentory
-# 3rd line: LF ; Line feed is mandantory between messages
-#
-# 'lang_en_??.txt'
-# 1st line: '#MSG_'<some text>' c='<max chars in a column>' r='<max rows> ; '#MSG' is mandentory while 'c=' and 'r=' aren't but should be there
-# 2nd line: '"'<origin message used in the source code>'"' ; '"' double quotes at the beginning and end of message are mandentory
-# 3rd line: '"'<translated message>'"' ; '"' double quotes at the beginning and end of message are mandentory
-# 4th line: LF ; Line feed is mandantory between messages
-#
-"""Check lang files."""
 from argparse import ArgumentParser
-from traceback import print_exc
 from sys import stdout, stderr, exit
+import codecs
+import polib
 import textwrap
 import re
+import os
+
+from lib import charset as cs
+from lib.io import load_map
+
+COLORIZE = (stdout.isatty() and os.getenv("TERM", "dumb") != "dumb") or os.getenv('NO_COLOR') == "0"
 
 def color_maybe(color_attr, text):
-    if stdout.isatty():
+    if COLORIZE:
         return '\033[0;' + str(color_attr) + 'm' + text + '\033[0m'
     else:
         return text
@@ -107,270 +102,231 @@ def highlight_trailing_white(text):
         return ret
 
 def wrap_text(text, cols):
-    # wrap text
+    ret = []
-    ret = list(textwrap.TextWrapper(width=cols).wrap(text))
+    for line in text.split('\n'):
-    if len(ret):
+        # wrap each input line in text individually
-        # add back trailing whitespace
+        tmp = list(textwrap.TextWrapper(width=cols).wrap(line))
-        ret[-1] += ' ' * (len(text) - len(text.rstrip()))
+        if len(ret):
+            # add back trailing whitespace
+            tmp[-1] += ' ' * (len(text) - len(text.rstrip()))
+        ret.extend(tmp)
     return ret
 
-def unescape(text):
-    if '\\' not in text:
-        return text
-    return text.encode('ascii').decode('unicode_escape')
-
 def ign_char_first(c):
     return c.isalnum() or c in {'%', '?'}
 
 def ign_char_last(c):
     return c.isalnum() or c in {'.', "'"}
 
-def parse_txt(lang, no_warning, warn_empty, information, import_check):
+def check_translation(entry, msgids, is_pot, no_warning, no_suggest, warn_empty, warn_same, information):
-    """Parse txt file and check strings to display definition."""
+    """Check strings to display definition."""
-    if lang == "en":
-        file_path = "lang_en.txt"
-    else:
-        if import_check:
-            file_path = "po/new/lang_en_%s.txt" % lang
-        else:
-            file_path = "lang_en_%s.txt" % lang
 
-    print(green("Start %s lang-check" % lang))
+    # fetch/decode entry for easy access
+    meta = entry.comment.split('\n', 1)[0]
+    source = entry.msgid
+    translation = entry.msgstr
+    line = entry.linenum
+    known_msgid = msgids is None or source in msgids
+    errors = 0
 
-    lines = 0
+    # Check comment syntax (non-empty and include a MSG id)
-    with open(file_path) as src:
+    if known_msgid or warn_empty:
-        while True:
+        if len(meta) == 0:
-            message = src.readline()
+            print(red("[E]: Translation doesn't contain any comment metadata on line %d" % line))
-            lines += 1
+            return False
-            #print(message) #Debug
+        if not meta.startswith('MSG'):
-            #check syntax 1st line starts with `#MSG`
+            print(red("[E]: Critical syntax error: comment doesn't start with MSG on line %d" % line))
-            if (message[0:4] != '#MSG'):
+            print(red(" comment: " + meta))
-                print(red("[E]: Critical syntax error: 1st line doesn't start with #MSG on line %d" % lines))
+            return False
-                print(red(message))
-                exit(1)
-            #Check if columns and rows are defined
-            comment = message.split(' ')
-            #Check if columns and rows are defined
-            cols = None
-            rows = None
-            for item in comment[1:]:
-                key, val = item.split('=')
-                if key == 'c':
-                    cols = int(val)
-                    #print ("c=",cols) #Debug
-                elif key == 'r':
-                    rows = int(val)
-                    #print ("r=",rows) #Debug
-                else:
-                    raise RuntimeError(
-                        "Unknown display definition %s on line %d" %
-                        (' '.join(comment), lines))
-            if cols is None and rows is None:
-                if not no_warning:
-                    print(yellow("[W]: No display definition on line %d" % lines))
-                cols = len(source)     # propably fullscreen
-            if rows is None:
-                rows = 1
-            elif rows > 1 and cols != 20:
-                print(yellow("[W]: Multiple rows with odd number of columns on line %d" % lines))
 
-            #Wrap text to 20 chars and rows
+    # Check if columns and rows are defined
-            source = src.readline()[:-1] #read whole line
+    tokens = meta.split(' ')
-            lines += 1
+    cols = None
-            #check if 2nd line of origin message beginns and ends with " double quote
+    rows = None
-            if (source[0]!="\""):
+    for item in tokens[1:]:
-                print(red('[E]: Critical syntax error: Missing " double quotes at beginning of message in source on line %d' % lines))
+        try:
-                print(red(source))
+            key, val = item.split('=')
-                exit(1)
+            if key == 'c':
-            if (source[-1]=="\""):
+                cols = int(val)
-                source = source.strip('"') #remove " double quotes from message 
+            elif key == 'r':
+                rows = int(val)
             else:
-                print(red('[E]: Critical syntax error: Missing " double quotes at end of message in source on line %d' % lines))
+                raise ValueError
-                print(red(source))
+        except ValueError:
-                exit(1)
+            print(red("[E]: Invalid display definition on line %d" % line))
-            #print(source) #Debug
+            print(red(" definition: " + meta))
-            if lang != "en":
+            return False
-                translation = src.readline()[:-1]#read whole line
-                lines += 1
-                #check if 3rd line of translation message beginns and ends with " double quote
-                if (translation[0]!="\""):
-                    print(red('[E]: Critical syntax error: Missing " double quotes at beginning of message in translation on line %d' % lines))
-                    print(red(translation))
-                    exit(1)
-                if (translation[-1]=="\""):
-                    #print ("End ok")
-                    translation = translation.strip('"') #remove " double quote from message
-                else:
-                    print(red('[E]: Critical syntax error: Missing " double quotes at end of message in translation on line %d' % lines))
-                    print(red(translation))
-                    exit(1)
-                #print(translation) #Debug
-                if translation == '\\x00':
-                    # crude hack to handle intentionally-empty translations
-                    translation = ''
-                #check if source is ascii only
-            if source.isascii() == False:
-                print(red('[E]: Critical syntax: Non ascii chars found on line %d' % lines))
-                print(red(source))
-                exit(1)
-            #check if translation is ascii only
-            if lang != "en":
-                if translation.isascii() == False:
-                    print(red('[E]: Critical syntax: Non ascii chars found on line %d' % lines))
-                    print(red(translation))
-                    exit(1)
 
-            # handle backslash sequences
+    if cols is None and rows is None:
-            source = unescape(source)
+        if not no_warning and known_msgid:
-            if lang != "en":
+            errors += 1
-                translation = unescape(translation)
+            print(yellow("[W]: No usable display definition on line %d" % line))
-            
+        # probably fullscreen, guess from the message length to continue checking
-            #print (translation) #Debug
+        cols = len(source)
-            wrapped_source = wrap_text(source, cols)
+    if rows is None:
-            rows_count_source = len(wrapped_source)
+        rows = 1
-            if lang != "en":
+    elif rows > 1 and cols != 20:
-                wrapped_translation = wrap_text(translation, cols)
+        errors += 1
-                rows_count_translation = len(wrapped_translation)
+        print(yellow("[W]: Multiple rows with odd number of columns on line %d" % line))
 
-            # Check for potential errors in the definition
+    # Check if translation contains unsupported characters
-            if not no_warning:
+    invalid_char = cs.translation_check(cs.unicode_to_source(translation))
-                # Incorrect number of rows/cols on the definition
+    if invalid_char is not None:
-                if rows == 1 and (len(source) > cols or rows_count_source > rows):
+        print(red('[E]: Critical syntax: Unhandled char %s found on line %d' % (repr(invalid_char), line)))
-                    print(yellow('[W]: Source text longer than %d cols as defined on line %d:' % (cols, lines)))
+        print(red(' translation: ' + translation))
-                    print_ruler(4, cols);
+        return False
-                    print_truncated(source, cols)
-                    print()
-                elif rows_count_source > rows:
-                    print(yellow('[W]: Wrapped source text longer than %d rows as defined on line %d:' % (rows, lines)))
-                    print_ruler(6, cols);
-                    print_wrapped(wrapped_source, rows, cols)
-                    print()
 
-                # Missing translation
+    # Pre-process the translation to translated characters for a correct preview and length check
-                if lang != "en":
+    translation = cs.trans_replace(translation)
-                    if len(translation) == 0 and (warn_empty or rows > 1):
-                        if rows == 1:
-                            print(yellow("[W]: Empty translation for \"%s\" on line %d" % (source, lines)))
-                        else:
-                            print(yellow("[W]: Empty translation on line %d" % lines))
-                            print_ruler(6, cols);
-                            print_wrapped(wrapped_source, rows, cols)
-                            print()
 
+    wrapped_source = wrap_text(source, cols)
+    rows_count_source = len(wrapped_source)
+    wrapped_translation = wrap_text(translation, cols)
+    rows_count_translation = len(wrapped_translation)
 
-                    # Check for translation lenght
+    # Incorrect number of rows/cols on the definition
-                    if (rows_count_translation > rows) or (rows == 1 and len(translation) > cols):
+    if rows == 1 and (len(source) > cols or rows_count_source > rows):
-                        print(red('[E]: Text is longer than definition on line %d: cols=%d rows=%d (rows diff=%d)'
+        errors += 1
-                                % (lines, cols, rows, rows_count_translation-rows)))
+        print(yellow('[W]: Source text longer than %d cols as defined on line %d:' % (cols, line)))
-                        print_source_translation(source, translation,
+        print_ruler(4, cols);
-                                                wrapped_source, wrapped_translation,
+        print_truncated(source, cols)
-                                                rows, cols)
+        print()
+    elif rows_count_source > rows:
+        errors += 1
+        print(yellow('[W]: Wrapped source text longer than %d rows as defined on line %d:' % (rows, line)))
+        print_ruler(6, cols);
+        print_wrapped(wrapped_source, rows, cols)
+        print()
 
-                    # Different count of % sequences
+    # All further checks are against the translation
-                    if source.count('%') != translation.count('%') and len(translation) > 0:
+    if is_pot:
-                        print(red('[E]: Unequal count of %% escapes on line %d:' % (lines)))
+        return (errors == 0)
-                        print_source_translation(source, translation,
-                                                wrapped_source, wrapped_translation,
-                                                rows, cols)
 
-                    # Different first/last character
+    # Missing translation
-                    if not no_warning and len(source) > 0 and len(translation) > 0:
+    if len(translation) == 0 and (known_msgid or warn_empty):
-                        source_end = source.rstrip()[-1]
+        errors += 1
-                        translation_end = translation.rstrip()[-1]
+        if rows == 1:
-                        start_diff = not (ign_char_first(source[0]) and ign_char_first(translation[0])) and source[0] != translation[0]
+            print(yellow("[W]: Empty translation for \"%s\" on line %d" % (source, line)))
-                        end_diff = not (ign_char_last(source_end) and ign_char_last(translation_end)) and source_end != translation_end
+        else:
-                        if start_diff or end_diff:
+            print(yellow("[W]: Empty translation on line %d" % line))
-                            if start_diff:
+            print_ruler(6, cols);
-                                print(yellow('[W]: Differing first punctuation character (%s => %s) on line %d:' % (source[0], translation[0], lines)))
+            print_wrapped(wrapped_source, rows, cols)
-                            if end_diff:
+            print()
-                                print(yellow('[W]: Differing last punctuation character (%s => %s) on line %d:' % (source[-1], translation[-1], lines)))
-                            print_source_translation(source, translation,
-                                                    wrapped_source, wrapped_translation,
-                                                    rows, cols)
-                    if not no_warning and source == translation:
-                        print(yellow('[W]: Translation same as origin on line %d:' %lines))
-                        print_source_translation(source, translation,
-                                                wrapped_source, wrapped_translation,
-                                                rows, cols)
-                    #elif information:
-                    #    print(green('[I]: %s' % (message)))
-                    #    print_source_translation(source, translation,
-                    #                            wrapped_source, wrapped_translation,
-                    #                            rows, cols)
 
+    # Check for translation lenght
+    if (rows_count_translation > rows) or (rows == 1 and len(translation) > cols):
+        errors += 1
+        print(red('[E]: Text is longer than definition on line %d: cols=%d rows=%d (rows diff=%d)'
+                % (line, cols, rows, rows_count_translation-rows)))
+        print_source_translation(source, translation,
+                                wrapped_source, wrapped_translation,
+                                rows, cols)
 
-                    # Short translation
+    # Different count of % sequences
-                    if not no_warning and len(source) > 0 and len(translation) > 0:
+    if source.count('%') != translation.count('%') and len(translation) > 0:
-                        if len(translation.rstrip()) < len(source.rstrip()) / 2:
+        errors += 1
-                            print(yellow('[W]: Short translation on line %d:' % (lines)))
+        print(red('[E]: Unequal count of %% escapes on line %d:' % (line)))
-                            print_source_translation(source, translation,
+        print_source_translation(source, translation,
-                                                    wrapped_source, wrapped_translation,
+                                wrapped_source, wrapped_translation,
-                                                    rows, cols)
+                                rows, cols)
-                    #elif information:
-                    #    print(green('[I]: %s' % (message)))
-                    #    print_source_translation(source, translation,
-                    #                            wrapped_source, wrapped_translation,
-                    #                            rows, cols)
 
-                    # Incorrect trailing whitespace in translation
+    # Different first/last character
-                    if not no_warning and len(translation) > 0 and \
+    if not no_suggest and len(source) > 0 and len(translation) > 0:
-                     (source.rstrip() == source or (rows == 1 and len(source) == cols)) and \
+        source_end = source.rstrip()[-1]
-                     translation.rstrip() != translation and \
+        translation_end = translation.rstrip()[-1]
-                     (rows > 1 or len(translation) != len(source)):
+        start_diff = not (ign_char_first(source[0]) and ign_char_first(translation[0])) and source[0] != translation[0]
-                        print(yellow('[W]: Incorrect trailing whitespace for translation on line %d:' % (lines)))
+        end_diff = not (ign_char_last(source_end) and ign_char_last(translation_end)) and source_end != translation_end
-                        source = highlight_trailing_white(source)
+        if start_diff or end_diff:
-                        translation = highlight_trailing_white(translation)
+            if start_diff:
-                        wrapped_translation = highlight_trailing_white(wrapped_translation)
+                print(yellow('[S]: Differing first punctuation character (%s => %s) on line %d:' % (source[0], translation[0], line)))
-                        print_source_translation(source, translation,
+            if end_diff:
-                                                wrapped_source, wrapped_translation,
+                print(yellow('[S]: Differing last punctuation character (%s => %s) on line %d:' % (source[-1], translation[-1], line)))
-                                                rows, cols)
+            print_source_translation(source, translation,
-                    elif information:
+                                    wrapped_source, wrapped_translation,
-                        print(green('[I]: %s' % (message)))
+                                    rows, cols)
-                        print_source_translation(source, translation,
+    if not no_suggest and source == translation and (warn_same or len(source.split(' ', 1)) > 1):
-                                                wrapped_source, wrapped_translation,
+        print(yellow('[S]: Translation same as original on line %d:' %line))
-                                                rows, cols)
+        print_source_translation(source, translation,
+                                wrapped_source, wrapped_translation,
+                                rows, cols)
 
+    # Short translation
+    if not no_suggest and len(source) > 0 and len(translation) > 0:
+        if len(translation.rstrip()) < len(source.rstrip()) / 2:
+            print(yellow('[S]: Short translation on line %d:' % (line)))
+            print_source_translation(source, translation,
+                                    wrapped_source, wrapped_translation,
+                                    rows, cols)
 
-            delimiter = src.readline()
+    # Incorrect trailing whitespace in translation
-            lines += 1
+    if not no_warning and len(translation) > 0 and \
-            if ("" == delimiter):
+     (source.rstrip() == source or (rows == 1 and len(source) == cols)) and \
-                break
+     translation.rstrip() != translation and \
-            elif len(delimiter) != 1:  # empty line
+     (rows > 1 or len(translation) != len(source)):
-                print(red('[E]: Critical Syntax error: Missing empty line between messages between lines: %d and %d' % (lines-1,lines)))
+        errors += 1
-                break
+        print(yellow('[W]: Incorrect trailing whitespace for translation on line %d:' % (line)))
-    print(green("End %s lang-check" % lang))
+        source = highlight_trailing_white(source)
+        translation = highlight_trailing_white(translation)
+        wrapped_translation = highlight_trailing_white(wrapped_translation)
+        print_source_translation(source, translation,
+                                wrapped_source, wrapped_translation,
+                                rows, cols)
+
+    # show the information
+    if information and errors == 0:
+        print(green('[I]: %s' % (meta)))
+        print_source_translation(source, translation,
+                                wrapped_source, wrapped_translation,
+                                rows, cols)
+    return (errors == 0)
 
 
 def main():
     """Main function."""
-    parser = ArgumentParser(
+    parser = ArgumentParser(description=__doc__)
-        description=__doc__,
+    parser.add_argument("po", help="PO file to check")
-        usage="%(prog)s lang")
-    parser.add_argument(
-        "lang", nargs='?', default="en", type=str,
-        help="Check lang file (en|cs|da|de|es|fr|hr|hu|it|lb|lt|nl|no|pl|ro|sk|sl|sv)")
     parser.add_argument(
         "--no-warning", action="store_true",
         help="Disable warnings")
     parser.add_argument(
-        "--warn-empty", action="store_true",
+        "--no-suggest", action="store_true",
-        help="Warn about empty translations")
+        help="Disable suggestions")
+    parser.add_argument(
+        "--pot", action="store_true",
+        help="Do not check translations")
     parser.add_argument(
         "--information", action="store_true",
         help="Output all translations")
+    parser.add_argument("--map",
+        help="Provide a map file to suppress warnings about unused translations")
     parser.add_argument(
-        "--import-check", action="store_true",
+        "--warn-empty", action="store_true",
-        help="Check import file and save informational to file")
+        help="Warn about empty definitions and translations even if unused")
+    parser.add_argument(
+        "--warn-same", action="store_true",
+        help="Warn about one-word translations which are identical to the source")
 
+    # load the translations
     args = parser.parse_args()
-    try:
+    if not os.path.isfile(args.po):
-        parse_txt(args.lang, args.no_warning, args.warn_empty, args.information, args.import_check)
+        print("{}: file does not exist or is not a regular file".format(args.po), file=stderr)
-        return 0
-    except Exception as exc:
-        print_exc()
-        parser.error("%s" % exc)
         return 1
 
+    # load the symbol map to supress empty (but unused) translation warnings
+    msgids = None
+    if args.map:
+        msgids = set()
+        for sym in load_map(args.map):
+            if type(sym['data']) == bytes:
+                msgid = cs.source_to_unicode(codecs.decode(sym['data'], 'unicode_escape', 'strict'))
+                msgids.add(msgid)
+
+    # check each translation in turn
+    status = True
+    for translation in polib.pofile(args.po):
+        status &= check_translation(translation, msgids, args.pot, args.no_warning, args.no_suggest,
+                                    args.warn_empty, args.warn_same, args.information)
+    return 0 if status else os.EX_DATAERR
 
 if __name__ == "__main__":
     exit(main())

lang/lang-check.sh

@@ -1,75 +0,0 @@
-#!/bin/bash
-#
-# lang_check.sh - multi-language support script
-#  check lang_xx.bin (language binary file)
-#
-# Input files:
-#  lang_$1.bin
-#  lang_en.txt or lang_en_$1.txt
-#  
-#
-
-#set 'cz'
-
-#dictionary txt file
-fn_t=lang_en_$1.txt
-if [ "$1" = "en" ]; then fn_t=lang_en.txt; fi
-#binary file to check
-fn_b=lang_$1.bin
-
-#check txt dictionary file
-echo -n "dictionary file: $fn_t"
-if [ -e $fn_t ]; then echo " - OK"; else echo " - Not found!"; exit 1; fi
-
-#create lang_xx.tmp - different processing for 'en' language
-if [ "$1" = "en" ]; then
- #remove comments and empty lines
- cat lang_en.txt | sed '/^$/d;/^#/d'
-else
- #remove comments and empty lines, print lines with translated text only
- cat lang_en_$1.txt | sed '/^$/d;/^#/d' | sed -n 'n;p'
-fi | sed 's/^\"\\x00\"$/\"\"/' > lang_$1.tmp
-
-count_txt=$(grep -c '^"' lang_$1.tmp)
-
-echo -n "language bin file: $fn_b"
-if [ -e $fn_b ]; then echo " - OK"; else echo " - Not found!"; exit 1; fi
-
-#read header and convert to hex
-header=$(dd if=$fn_b bs=1 count=16 2>/dev/null | xxd | cut -c11-49 | sed 's/\([0-9a-f][0-9a-f]\)[\ ]*/\1 /g')
-echo "header='$header'"
-magic=0x$(echo $header | tr -d ' ' | cut -c1-8)
-echo "magic='$magic'"
-size=$(echo $header | tr -d ' ' | cut -c9-12)
-size=0x${size:2:2}${size:0:2}
-echo "size='$size' ($(($size)))"
-count=$(echo $header | tr -d ' ' | cut -c13-16)
-count=0x${count:2:2}${count:0:2}
-echo "count='$count' ($(($count)))"
-o=0
-l=0
-#create lang_xx_1.tmp (temporary text file from binary data)
-(dd if=$fn_b bs=1 count=$((2*$count)) skip=16 2>/dev/null | xxd | cut -c11-49 | tr ' ' "\n" |\
- sed 's/\([0-9a-f][0-9a-f]\)\([0-9a-f][0-9a-f]\)/\2\1 /g;/^$/d'; printf "%04x\n" $(($size)) ) |\
- while read offs; do
-  if [ $o -ne 0 ]; then
-   l=$((0x$offs - $o))
-   echo -n '"'
-   dd if=$fn_b bs=1 count=$((l-1)) skip=$o 2>/dev/null
-   echo '"'
-  fi
-  o=$((0x$offs))
- done > lang_$1_1.tmp
-#create lang_xx_2.tmp (temporary text file from dictionary)
-cat lang_$1.tmp | sed 's/^\"/printf \"\\x22/;s/"$/\\x22\\x0a\"/' | sh >lang_$1_2.tmp
-#compare temporary files
-diff -a lang_$1_1.tmp lang_$1_2.tmp >lang_$1_check.dif
-dif=$(cat lang_$1_check.dif)
-if [ -z "$dif" ]; then
- echo 'binary data OK'
-else
- echo 'binary data NG!'
-fi
-
-read -t 5
-exit

lang/lang-clean.sh

@@ -1,106 +0,0 @@
-#!/bin/bash
-#
-# Version 1.0.1 Build 13
-#
-# clean.sh - multi-language support script
-#  Remove all language output files from lang folder.
-#
-#############################################################################
-# Change log:
-#  1 Nov. 2018, XPila,      Initial
-# 18 Oct. 2018, XPila,      New lang, arduino 1.8.5 - fw-clean.sh and lang-clean.sh fix
-# 25 Oct. 2018, XPila,      New lang - fixed french langcode and comparsion in lang-clean script
-# 10 Dec. 2018, jhoblitt,   make all shell scripts executable
-# 26 Jul. 2019, leptun,     Fix shifted languages. Use \n and \x0a
-# 14 Sep. 2019, 3d-gussner, Prepare adding new language
-# 01 Mar. 2021, 3d-gussner, Move `Dutch` language parts
-# 22 Mar. 2021, 3d-gussner, Move Dutch removing part to correct loaction
-# 21 Dec. 2021, 3d-gussner, Use one config file for all languages
-# 03 Jan. 2022, 3d-gussner, Cleanup outdated code
-# 11 Jan. 2022, 3d-gussner, Also remove temporally files which have been
-#                           generated for message and size count comparison
-#                           Added version and Change log
-#                           colored output
-#                           Add Community language support
-#                           Use `git rev-list --count HEAD lang-clean.sh`
-#                           to get Build Nr
-# 25 Jan. 2022, 3d-gussner, clean up lang-import.sh temproray files
-# 14 Feb. 2022, 3d-gussner, Fix single language run without config.sh OK
-# 16 Mar. 2022, 3d-gussner, Cleanup `output-layout/sorted-??.txt` 
-#############################################################################
-result=0
-
-rm_if_exists()
-{
- if [ -e $1 ]; then
-  echo -n "$(tput sgr0) removing $(tput setaf 3)'$1'$(tput sgr0)..." >&2
-  if rm $1; then
-   echo "$(tput setaf 2)OK$(tput sgr0)" >&2
-  else
-   echo "$(tput setaf 1)NG!$(tput sgr0)" >&2
-   result=1
-  fi
- fi
-}
-
-clean_lang()
-{
- if [ "$1" = "en" ]; then
-  rm_if_exists lang_$1.tmp
-  rm_if_exists lang_$1.cnt
-  rm_if_exists lang_$1.max
- else
-  rm_if_exists lang_$1.tmp
-  rm_if_exists lang_en_$1.tmp
-  rm_if_exists lang_en_$1.dif
-  rm_if_exists lang_$1.ofs
-  rm_if_exists lang_$1.txt
-  rm_if_exists po/new/$1_new.po
-  rm_if_exists po/new/$1.mo
-  rm_if_exists po/new/$1_filtered.po
-  rm_if_exists po/new/lang_en_$1.txt
-  rm_if_exists po/new/output-layout-$1.txt
-  rm_if_exists po/new/output-sorted-$1.txt
- fi
- rm_if_exists lang_$1_check.dif
- rm_if_exists lang_$1.bin
- rm_if_exists lang_$1.dat
- rm_if_exists lang_$1_1.tmp
- rm_if_exists lang_$1_2.tmp
- rm_if_exists po/new/nonascii.txt
-
-}
-
-echo "$(tput setaf 2)lang-clean.sh started$(tput sgr0)" >&2
-#Clean English
- clean_lang en
-
-#Clean languages
-echo "lang-clean languages:$(tput setaf 2)$LANGUAGES$(tput sgr0)" >&2
-if [ -e $1 ]; then
-# Config:
-  if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-  if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
-
-  if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
-    LANGUAGES+=" $COMMUNITY_LANGUAGES"
-  fi
-
- for lang in $LANGUAGES; do
-  clean_lang $lang
- done
-else
-  clean_lang $1
-fi
-
-if [ $result -eq 0 ]; then
- echo "$(tput setaf 2) lang-clean.sh with success$(tput sgr0)" >&2
-else
- echo "$(tput setaf 1) lang-clean.sh with errors!$(tput sgr0)" >&2
-fi
-
-case "$-" in
- *i*) echo "press enter key" >&2; read ;;
-esac
-
-exit $result

lang/lang-community.sh

@@ -1,53 +0,0 @@
-#!/bin/sh
-#
-# lang-community.sh - Community language support script
-# Check community languages are defined in `config.h`
-#
-
-# Root path
-if [ -z "$ROOT_PATH" ]; then
-    export ROOT_PATH=".."
-fi
-
-# Check community language NL = Dutch
-COMMUNITY_LANG_GROUP1_NL=$(grep --max-count=1 "^#define COMMUNITY_LANG_GROUP1_NL" $ROOT_PATH/Firmware/config.h| cut -d '_' -f3 |cut -d ' ' -f1)
-export NL=$COMMUNITY_LANG_GROUP1_NL
-
-# Use the lines below as a template and replace 'QR' and 'new language'
-# Check comminity language QR = new language
-#COMMUNITY_LANG_QR=$(grep --max-count=1 "^#define COMMUNITY_LANG_QR" $ROOT_PATH/Firmware/config.h| cut -d '_' -f3 |cut -d ' ' -f1)
-#export QR=$COMMUNITY_LANG_QR
-
-#startup message
-echo "lang-community.sh started" >&2
-echo -n " Source code path: " >&2
-if [ -e $ROOT_PATH ]; then echo 'OK' >&2; else echo 'NG!' >&2; _err=1; fi
-
-echo " Found: " >&2
-if [ "$COMMUNITY_LANG_GROUP1_NL" = "NL" ]; then
-    echo "  $COMMUNITY_LANG_GROUP1_NL" >&2
-    echo
-    ./lang-build.sh nl
-fi
-
-# Use the 5 lines below as a template and replace 'QR' and 'qr'
-#if [ "$COMMUNITY_LANG_QR" = "QR" ]; then
-#    echo "  $COMMUNITY_LANG_QR" >&2
-#    echo
-#    ./lang-build.sh qr
-#fi
-
-
-#exiting function
-finish()
-{
- if [ $1 -eq 0 ]; then
-  echo "lang-community.sh finished with success" >&2
- else
-  echo "lang-community.sh finished with errors!" >&2
- fi
- echo 
- exit $1
-}
-
-finish 0

lang/lang-export.sh

@@ -1,275 +0,0 @@
-#!/bin/bash
-#
-# Version 1.0.1 Build 32
-#
-# lang-export.sh - multi-language support script
-#  for generating /lang/po/Firmware_xx.po
-#
-#############################################################################
-# Change log:
-#  9 Nov. 2018, XPila,      Initial
-# 10 Dec. 2018, jhoblitt,   make all shell scripts executable
-# 14 Sep. 2019, 3d-gussner, Prepare adding new language
-#  6 Sep. 2019, DRracer,    change to bash
-#  1 Mar. 2019, 3d-gussner, Move `Dutch` language parts
-#                           Add templates for future community languages
-# 17 Dec. 2021, 3d-gussner, Use one config file for all languages
-#                           Fix missing last translation
-# 21 Dec. 2021, 3d-gussner, Add Swedish, Danish, Slovanian, Hungarian,
-#                           Luxembourgish, Croatian
-#  3 Jan. 2022, 3d-gussner, Add Lithuanian
-#                           Cleanup outaded code
-# 11 Jan. 2022, 3d-gussner, Added version and Change log
-#                           colored output
-#                           Add Community language support
-#                           Use `git rev-list --count HEAD lang-export.sh`
-#                           to get Build Nr
-# 25 Jan. 2022, 3d-gussner, Replace German HD44780 A00 ROM 'äöüß' to UTF-8 'äöüß'
-# 14 Feb. 2022, 3d-gussner, Fix single language run without config.sh OK
-# 12 Mar. 2022, 3d-gussner, Update Norwegian replace umlaut and diacritics
-#                           Fix find community languages
-#                           Update Swedish replace umlaut and diacritics
-#                           Replace '.!? äöü' with '.!? ÄÖÜ' in German and Swedish
-#                           Replace '"äöü' with '"ÄÖÜ' in German and Swedish
-# 18 Mar. 2022, 3d-gussner, Add Swedish ` pa ` to ` på ` conversion
-#############################################################################
-
-echo "$(tput setaf 2)lang-export.sh started$(tput sgr 0)" >&2
-
-# relative path to source folder
-if [ -z "$SRCDIR" ]; then
-   SRCDIR=".."
-fi
-
-# selected language is 1st argument (cz, de, ...)
-LNG=$1
-
-# if no arguments, 'all' is selected (all po and also pot will be generated)
-if [ -z "$LNG" ]; then
-  LNG="all";
-# Config:
-  if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-  if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr 0)" >&2; exit 1; fi
-  if [ ! -z "$COMMUNITY_LANGUAGES" ]; then
-    LANGUAGES+=" $COMMUNITY_LANGUAGES"
-    echo $LANGUAGES>&2
-  fi
-  echo "$(tput setaf 2)lang-export languages:$LANGUAGES$(tput sgr 0)" >&2
-fi
-
-# if 'all' is selected, script will generate all po files and also pot file
-if [ "$LNG" = "all" ]; then
- ./lang-export.sh en
-  for lang in $LANGUAGES; do
-   ./lang-export.sh $lang
-  done
- exit 0
-fi
-
-# language code (iso639-1) is equal to LNG
-LNGISO=$LNG
-# exception for 'cz' (code='cs')
-if [ "$LNG" = "cz" ]; then LNGISO=cs; fi
-
-# po/pot creation/revision date
-DATE=$(date)
-
-# if 'en' is selected, generate pot instead of po
-if [ "$LNG" = "en" ]; then
- INFILE=lang_en.txt
- OUTFILE=po/Firmware.pot
- LNGNAME="English"
-else
- # language name in english
- LNGNAME=$(\
-  case "$LNG" in
-   *cz*) echo "Czech" ;;
-   *de*) echo "German" ;;
-   *es*) echo "Spanish" ;;
-   *fr*) echo "French" ;;
-   *it*) echo "Italian" ;;
-   *pl*) echo "Polish" ;;
-#Community language support
-#Dutch
-   *nl*) echo "Dutch" ;;
-#Swedish
-   *sv*) echo "Swedish" ;;
-#Norwegian
-   *no*) echo "Norwegian" ;;
-#Danish
-   *da*) echo "Danish" ;;
-#Slovak
-   *sk*) echo "Slovak" ;;   
-#Slovanian
-   *sl*) echo "Slovanian" ;;
-#Hugarian
-   *hu*) echo "Hugarian" ;;
-#Luxembourgish
-   *lb*) echo "Luxembourgish" ;;
-#Croatian
-   *hr*) echo "Croatian" ;;
-#Lithuanian
-   *lt*) echo "Lithuanian" ;;
-#Romanian
-   *ro*) echo "Romanian" ;;
-#Use the 2 lines below as a template and replace 'qr' and 'New language'
-##New language
-#   *qr*) echo "New language" ;;
-  esac)
- # unknown language - error
- if [ -z "LNGNAME" ]; then
-  echo "Invalid argument $(tput setaf 1)'$LNG'$(tput sgr 0).">&2
-  exit 1
- fi
- INFILE=lang_en_$LNG.txt
- OUTFILE=po/Firmware_$LNGISO.po
-fi
-
-# remove output file if exists
-if [ -e $OUTFILE ]; then rm -f -v $OUTFILE; fi
-
-#total strings
-CNTTXT=$(grep '^#' -c $INFILE)
-#not translated strings
-CNTNT=$(grep '^\"\\x00\"' -c $INFILE)
-echo " $(tput setaf 2)$CNTTXT$(tput sgr 0) texts, $(tput setaf 3)$CNTNT$(tput sgr 0) not translated" >&2
-
-# list .cpp, .c and .h files from source folder
-SRCFILES=$(ls "$SRCDIR/Firmware"/*.cpp "$SRCDIR/Firmware"/*.c "$SRCDIR/Firmware"/*.h)
-
-echo " selected language=$(tput setaf 2)$LNGNAME$(tput sgr 0)" >&2
-
-# write po/pot header
-(
- echo "# Translation of Prusa-Firmware into $LNGNAME."
- echo "#"
- echo 'msgid ""'
- echo 'msgstr ""'
- echo '"MIME-Version: 1.0\n"'
- echo '"Content-Type: text/plain; charset=UTF-8\n"'
- echo '"Content-Transfer-Encoding: 8bit\n"'
- echo '"Language: '$LNGISO'\n"'
- echo '"Project-Id-Version: Prusa-Firmware\n"'
- echo '"POT-Creation-Date: '$DATE'\n"'
- echo '"PO-Revision-Date: '$DATE'\n"'
- echo '"Language-Team: \n"'
- echo '"X-Generator: Poedit 2.0.7\n"'
- echo '"X-Poedit-SourceCharset: UTF-8\n"'
- echo '"Last-Translator: \n"'
- echo '"Plural-Forms: nplurals=3; plural=(n==1) ? 0 : (n>=2 && n<=4) ? 1 : 2;\n"'
- echo
-) >$OUTFILE
-
-#loop over all messages
-s0=''
-s1=''
-s2=''
-num=1
-(cat $INFILE | sed "s/\\\\/\\\\\\\\/g" | while read -r s; do
- #start debug
- #if [ "${s:0:1}" = "\"" ]; then
- # echo  >&2
- # echo "s = $s ." >&2
- # echo "s0 = $s0 ." >&2
- # echo "s1 = $s1 ." >&2
- #fi
- #end debug
- if [ "${s:0:1}" = "\"" ]; then
-  if [[ "${s0:0:1}" = "\"" || "$LNG" = "en" ]]; then
-   echo -ne "  processing $num of $CNTTXT\033[0K\r" >&2
-   # write po/pot item
-   (
-   if [ "$LNG" = "en" ]; then s1=$s0; s0=$s; fi
-    search=$(/bin/echo -e "$s0")
-    found=$(grep -m1 -n -F "$search" $SRCFILES | head -n1 | cut -f1-2 -d':' | sed "s/^.*\///")
-    echo "$s1" | sed 's/ c=0//;s/ r=0//;s/^#/# /'
-    #echo "$s1" | sed 's/ c=0//;s/ r=0//;s/^#/# /' >&2
-    echo "#: $found"
-    #echo "#: $found" >&2
-    /bin/echo -e "msgid $s0"
-    if [[ "$s" = "\"\\\\x00\"" || "$LNG" = "en" ]]; then
-     echo 'msgstr ""'
-    else
-     /bin/echo -e "msgstr $s"
-    fi
-    echo
-  )
-  num=$((num+1))
-  fi
- fi
- s1=$s0
- s0=$s
-done >>$OUTFILE) 2>&1
-
-#replace LF with CRLF
-sync
-sed -i 's/$/\r/' $OUTFILE
-
-#replace HD44780 A00 'äöüß' to UTF-8 'äöüß'
-if [[ "$LNG" = "de" || "$LNG" = "sv" ]]; then
-  #replace 'A00 ROM '"ä' with '"Ä' 
-  sed -i 's/"\\xe1/"\xc3\x84/g' $OUTFILE
-  #replace 'A00 ROM '"ü' with '"Ü'
-  sed -i 's/"\\xf5/"\xc3\x9c/g' $OUTFILE
-  #replace 'A00 ROM '"ö' with '"Ö'
-  sed -i 's/"\\xef/"\xc3\x96/g' $OUTFILE
-  #replace 'A00 ROM '. ä' with '. Ä' 
-  sed -i 's/\. \\xe1/. \xc3\x84/g' $OUTFILE
-  #replace 'A00 ROM '. ü' with '. Ü'
-  sed -i 's/\. \\xf5/. \xc3\x9c/g' $OUTFILE
-  #replace 'A00 ROM '. ö' with '. Ö'
-  sed -i 's/\. \\xef/. \xc3\x96/g' $OUTFILE
-  #replace 'A00 ROM '! ä' with '! Ä' 
-  sed -i 's/! \\xe1/! \xc3\x84/g' $OUTFILE
-  #replace 'A00 ROM '! ü' with '! Ü'
-  sed -i 's/! \\xf5/! \xc3\x9c/g' $OUTFILE
-  #replace 'A00 ROM '! ö' with '! Ö'
-  sed -i 's/! \\xef/! \xc3\x96/g' $OUTFILE
-  #replace 'A00 ROM '? ä' with '? Ä' 
-  sed -i 's/? \\xe1/? \xc3\x84/g' $OUTFILE
-  #replace 'A00 ROM '? ü' with '? Ü'
-  sed -i 's/? \\xf5/? \xc3\x9c/g' $OUTFILE
-  #replace 'A00 ROM '? ö' with '? Ö'
-  sed -i 's/? \\xef/? \xc3\x96/g' $OUTFILE
-  #replace 'A00 ROM 'ä' with 'ä' 
-  sed -i 's/\\xe1/\xc3\xa4/g' $OUTFILE
-  #replace 'A00 ROM 'ü' with 'ü'
-  sed -i 's/\\xf5/\xc3\xbc/g' $OUTFILE
-  #replace 'A00 ROM 'ö' with 'ö'
-  sed -i 's/\\xef/\xc3\xb6/g' $OUTFILE
-  #replace 'A00 ROM 'ß'' with 'ß'
-  sed -i 's/\\xe2/\xc3\x9f/g' $OUTFILE
-fi
-
-if [ "$LNG" = "no" ]; then
-  #replace often used words
-  #replace ' pa ' with ' på ' 
-  sed -i 's/\ pa / p\xc3\xa5 /g' $OUTFILE
-  #replace ' na ' with ' nå ' 
-  sed -i 's/\ na / n\xc3\xa5 /g' $OUTFILE
-  #replace '"Na ' with '"Nå ' 
-  sed -i 's/\"Na /"N\xc3\xa5 /g' $OUTFILE
-  #replace ' stal' with ' stål' 
-  sed -i 's/\ stal/ st\xc3\xa5l/g' $OUTFILE
-  #replace HD44780 A00 'äö' to UTF-8 'æø'
-  #replace 'A00 ROM ä' with 'æ' 
-  sed -i 's/\\xe1/\xc3\xa6/g' $OUTFILE
-  #replace 'A00 ROM ö' with 'ø'
-  sed -i 's/\\xef/\xc3\xb8/g' $OUTFILE
-fi
-
-if [ "$LNG" = "sv" ]; then
-  #replace often used words
-  #replace ' pa ' with ' på ' 
-  sed -i 's/\ pa / p\xc3\xa5 /g' $OUTFILE
-fi
-
-
-#replace HD44780 A00 'μ' to UTF-8 'μ'
-#replace 'A00 ROMμ' with ' μ'
-sed -i 's/\\xe4/\xce\xbc/g' $OUTFILE
-
-
-echo >&2
-echo "$(tput setaf 2)lang-export.sh finished$(tput sgr 0)">&2
-exit 0

lang/lang-extract.py

@@ -0,0 +1,305 @@
+#!/usr/bin/env python3
+import argparse
+import bisect
+import codecs
+import polib
+import regex
+import sys
+import lib.charset as cs
+
+def line_warning(path, line, msg):
+    print(f'{path}:{line}: {msg}', file=sys.stderr)
+
+def line_error(path, line, msg):
+    print(f'{path}:{line}: {msg}', file=sys.stderr)
+
+def entry_warning_locs(entries):
+    for msgid, data in entries:
+        print('   text: ' + repr(msgid), file=sys.stderr)
+        positions = ', '.join(map(lambda x: x[0] + ':' + str(x[1]), data['occurrences']))
+        print('     in: ' + positions, file=sys.stderr)
+
+def entries_warning(entries, msg):
+    print('warning: ' + msg, file=sys.stderr)
+    entry_warning_locs(entries)
+
+def entry_warning(entry, msg):
+    entries_warning([entry], msg)
+
+
+def newline_positions(source):
+    lines = [-1]
+    while True:
+        idx = source.find('\n', lines[-1] + 1)
+        if idx < 0:
+            break
+        lines.append(idx)
+    if lines[-1] != len(source) - 1:
+        lines.append(len(source) - 1)
+    return lines[1:]
+
+def index_to_line(index, lines):
+    return bisect.bisect_left(lines, index) + 1
+
+
+def extract_file(path, catalog, warn_skipped=False):
+    source = open(path).read()
+    newlines = newline_positions(source)
+
+    # match internationalized quoted strings
+    RE_START = r'\b (_[iI]|ISTR) \s* \('
+    RE_META = r'//// \s* ([^\n]*)$'
+
+    RE_I = fr'''
+        (?<!(?:/[/*]|^\s*\#) [^\n]*)  # not on a comment or preprocessor
+        {RE_START}                    # $1 ref type _i( or ISTR(
+        (?:
+          \s*
+          ("(?:[^"\\]|\\.)*")         # $2 quoted string (chunk)
+          (?:\s* {RE_META} )?         # $3 inline metadata
+        )+
+        \s* \)                        # )
+        (?:
+          (?:[^\n] (?!{RE_START}))*   # anything except another entry
+          {RE_META}                   # $5 final metadata
+        )?
+    '''
+
+    r_ref_type = 1
+    r_quoted_chunk = 2
+    r_inline_data = 3
+    r_eol_data = 5
+
+    for m in regex.finditer(RE_I, source, regex.M|regex.X):
+        # parse the text
+        line = index_to_line(m.start(0), newlines)
+
+        text = ""
+        for block in m.captures(r_quoted_chunk):
+            # remove quotes and unescape
+            block = block[1:-1]
+            block = codecs.decode(block, 'unicode-escape', 'strict')
+            block = cs.source_to_unicode(block)
+            text += block
+
+        # check if text is non-empty
+        if len(text) == 0:
+            line_warning(path, line, 'empty source text, ignored')
+            continue
+
+        data = set()
+        comments = set()
+        for n in [r_inline_data, r_eol_data]:
+            meta = m.group(n)
+            if meta is not None:
+                meta_parts = meta.split('//', 1)
+                data.add(meta_parts[0].strip())
+                if len(meta_parts) > 1:
+                    comments.add(meta_parts[1].strip())
+
+        # check if this message should be ignored
+        ignored = False
+        for meta in data:
+            if regex.search(r'\bIGNORE\b', meta) is not None:
+                ignored = True
+                break
+        if ignored:
+            if warn_skipped:
+                line_warning(path, line, 'skipping explicitly ignored translation')
+            continue
+
+        # extra message catalog name (if any)
+        cat_name = set()
+        for meta in data:
+            sm = regex.search(r'\b(MSG_\w+)', meta)
+            if sm is not None:
+                cat_name.add(sm.group(1))
+
+        # reference type annotation
+        ref_type = 'def' if m.group(r_ref_type) == 'ISTR' else 'ref'
+        if ref_type == 'def':
+            # ISTR definition: lookup nearby assignment
+            lineup_def = source[newlines[line-2]+1:m.end(r_ref_type)]
+            sm = regex.search(r'\b PROGMEM_(\S+) \s*=\s* ISTR $', lineup_def, regex.M|regex.X)
+            if sm is None:
+                line_warning(path, line, 'ISTR not used in an assignment')
+            elif sm.group(1) != 'I1':
+                line_warning(path, line, 'ISTR not used with PROGMEM_I1')
+
+        # append the translation to the catalog
+        pos = [(path, line)]
+        entry = catalog.get(text)
+        if entry is None:
+            catalog[text] = {'occurrences': set(pos),
+                             'data': data,
+                             'cat_name': cat_name,
+                             'comments': comments,
+                             'ref_type': set([ref_type])}
+        else:
+            entry['occurrences'] = entry['occurrences'].union(pos)
+            entry['data'] = entry['data'].union(data)
+            entry['cat_name'] = entry['cat_name'].union(cat_name)
+            entry['comments'] = entry['comments'].union(comments)
+            entry['ref_type'].add(ref_type)
+
+
+def extract_refs(path, catalog):
+    source = open(path).read()
+    newlines = newline_positions(source)
+
+    # match message catalog references to add backrefs
+    RE_CAT = r'''
+        (?<!(?:/[/*]|^\s*\#) [^\n]*)         # not on a comment or preprocessor
+        \b (?:_[TO]) \s* \( \s* (\w+) \s* \) # $1 catalog name
+    '''
+
+    for m in regex.finditer(RE_CAT, source, regex.M|regex.X):
+        line = index_to_line(m.start(0), newlines)
+        pos = [(path, line)]
+        cat_name = m.group(1)
+        found = False
+        defined = False
+        for entry in catalog.values():
+            if cat_name in entry['cat_name']:
+                entry['occurrences'] = entry['occurrences'].union(pos)
+                entry['ref_type'].add('ref')
+                found = True
+                if 'def' in entry['ref_type']:
+                    defined = True
+        if not found:
+            line_error(path, line, f'{cat_name} not found')
+        elif not defined:
+            line_error(path, line, f'{cat_name} referenced but never defined')
+
+
+def check_entries(catalog, warn_missing, warn_same_line):
+    cat_entries = {}
+
+    for entry in catalog.items():
+        msgid, data = entry
+
+        # ensure we have at least one name
+        if len(data['cat_name']) == 0 and warn_missing:
+            entry_warning(entry, 'missing MSG identifier')
+
+        # ensure references are being defined
+        if data['ref_type'] == set(['def']):
+            if len(data['cat_name']) == 0:
+                if warn_missing:
+                    entry_warning(entry, 'entry defined, but never used')
+            else:
+                id_name = next(iter(data['cat_name']))
+                entry_warning(entry, f'{id_name} defined, but never used')
+
+        # check custom characters
+        invalid_char = cs.source_check(msgid)
+        if invalid_char is not None:
+            entry_warning(entry, 'source contains unhandled custom character ' + repr(invalid_char))
+
+        tokens = []
+        for meta in data['data']:
+            tokens.extend(regex.split(r'\s+', meta))
+        seen_keys = set()
+        for token in tokens:
+            if len(token) == 0:
+                continue
+
+            # check metadata syntax
+            if regex.match(r'[cr]=\d+$', token) is None and \
+               regex.match(r'MSG_[A-Z_0-9]+$', token) is None:
+                entry_warning(entry, 'bogus annotation: ' + repr(token))
+
+            # check for repeated keys
+            key = regex.match(r'([^=])+=', token)
+            if key is not None:
+                key_name = key.group(1)
+                if key_name in seen_keys:
+                    entry_warning(entry, 'repeated annotation: ' + repr(token))
+                else:
+                    seen_keys.add(key_name)
+
+            # build the inverse catalog map
+            if token.startswith('MSG_'):
+                if token not in cat_entries:
+                    cat_entries[token] = [entry]
+                else:
+                    cat_entries[token].append(entry)
+
+    # ensure the same id is not used in multiple entries
+    for cat_name, entries in cat_entries.items():
+        if len(entries) > 1:
+            entries_warning(entries, f'{cat_name} used in multiple translations')
+
+    if warn_same_line:
+        # build the inverse location map
+        entry_locs = {}
+        for entry in catalog.items():
+            msgid, data = entry
+            for loc in data['occurrences']:
+                if loc not in entry_locs:
+                    entry_locs[loc] = [loc]
+                else:
+                    entry_locs[loc].append(loc)
+
+        # check for multiple translations on the same location
+        for loc, entries in entry_locs.items():
+            if len(entries) > 1:
+                line_warning(loc[0], loc[1], f'line contains multiple translations')
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('-o', dest='pot', required=True, help='PO template output file')
+    ap.add_argument('--no-missing', action='store_true',
+                    help='Do not warn about missing MSG entries')
+    ap.add_argument('--warn-same-line', action='store_true',
+                    help='Warn about multiple translations on the same line')
+    ap.add_argument('--warn-skipped', action='store_true',
+                    help='Warn about explicitly ignored translations')
+    ap.add_argument('-s', '--sort', action='store_true',
+                    help='Sort output catalog')
+    ap.add_argument('file', nargs='+', help='Input files')
+    args = ap.parse_args()
+
+    # extract strings
+    catalog = {}
+    for path in args.file:
+        extract_file(path, catalog, warn_skipped=args.warn_skipped)
+
+    # process backreferences in a 2nd pass
+    for path in args.file:
+        extract_refs(path, catalog)
+
+    # check the catalog entries
+    check_entries(catalog, warn_missing=not args.no_missing, warn_same_line=args.warn_same_line)
+
+    # write the output PO template
+    po = polib.POFile()
+    po.metadata = {
+        'Language': 'en',
+        'MIME-Version': '1.0',
+        'Content-Type': 'text/plain; charset=utf-8',
+        'Content-Transfer-Encoding': '8bit'}
+
+    messages = catalog.keys()
+    if args.sort:
+        messages = sorted(messages)
+    for msgid in messages:
+        data = catalog[msgid]
+        comment = ', '.join(data['data'])
+        if len(data['comments']):
+            comment += '\n' + '\n'.join(data['comments'])
+        occurrences = data['occurrences']
+        if args.sort:
+            occurrences = list(sorted(occurrences))
+        po.append(
+            polib.POEntry(
+                msgid=msgid,
+                comment=comment,
+                occurrences=occurrences))
+
+    po.save(args.pot)
+    return 0
+
+if __name__ == '__main__':
+    exit(main())

lang/lang-import.sh

@@ -1,515 +0,0 @@
-#!/bin/bash
-#
-# Version 1.0.1 Build 46
-#
-# lang-import.sh - multi-language support script
-#  for importing translated xx.po
-#
-#############################################################################
-# Change log:
-#  9 Nov. 2018, XPila,      Initial
-# 21 Nov. 2018, XPila,      fix - replace '\n' with space in all languages
-# 10 Dec. 2018, jhoblitt,   make all shell scripts executable
-# 21 Aug. 2019, 3d-gussner, Added "All" argument and it is default in nothing is chosen
-#                           Added few German/French diacritical characters
-#  6 Sep. 2019, DRracer,    change to bash
-# 14 Sep. 2019, 3d-gussner, Prepare adding new language
-#  1 Mar. 2019, 3d-gussner, Move `Dutch` language parts
-#                           Add templates for future community languages
-# 17 Dec. 2021, 3d-gussner, Use one config file for all languages
-#                           Fix missing last translation
-#                           Add counter
-#                           replace two double quotes with `\x00`
-# 21 Dec. 2021, 3d-gussner, Add Swedish, Danish, Slovanian, Hungarian,
-#                           Luxembourgish, Croatian
-#  3 Jan. 2022, 3d-gussner, Add Lithuanian
-#                           Cleanup outaded code
-# 11 Jan. 2022, 3d-gussner, Added version and Change log
-#                           colored output
-#                           Add Community language support
-#                           Use `git rev-list --count HEAD lang-import.sh`
-#                           to get Build Nr
-# 14 Jan. 2022, 3d-gussner, Replace German UTF-8 'äöÿÿ' to HD44780 A00 ROM 'äöÿÿ'
-# 28 Jan. 2022, 3d-gussner, Run lang-check and output `xx-output.txt` file to review
-#                           translations
-#                           new argruments `--information` `--import-check`
-# 11 Jan. 2022, ingbrzy,    Add Slovak letters
-# 11 Feb. 2022, 3d-gussner, Change to python3
-# 14 Feb. 2022, 3d-gussner, Replace non-block space with space
-#                           Fix single language run without config.sh OK
-# 12 Mar. 2022, 3d-gussner, Update Norwegian replace umlaut and diacritics
-#                           Update Swedish umlaut and diacritics
-#############################################################################
-
-echo "$(tput setaf 2)lang-import.sh started$(tput sgr 0)" >&2
-
-LNG=$1
-# if no arguments, 'all' is selected (all po and also pot will be generated)
-if [ -z "$LNG" ]; then
-  LNG=all;
-# Config:
-  if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-  if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr 0)" >&2; exit 1; fi
-fi
-
-if [[ ! -z "$COMMUNITY_LANGUAGES" && "$LNG" = "all" ]]; then
-  LANGUAGES+=" $COMMUNITY_LANGUAGES"
-else
-  LANGUAGES="$LNG"
-fi
-echo "$(tput setaf 2)lang-import languages:$LANGUAGES$(tput sgr 0)" >&2
-
-# if 'all' is selected, script will generate all po files and also pot file
-if [ "$LNG" = "all" ]; then
-  for lang in $LANGUAGES; do
-   ./lang-import.sh $lang
-  done
- exit 0
-fi
-
-# language code (iso639-1) is equal to LNG
-LNGISO=$LNG
-# exception for 'cz' (code='cs')
-if [ "$LNG" = "cz" ]; then LNGISO=cs; fi
-
-# cd to input folder
-cd po/new
-
-# check if input file exists
-if ! [ -e $LNGISO.po ]; then
- echo "$(tput setaf 1)Input file $LNGISO.po not found!$(tput sgr 0)" >&2
- exit -1
-fi
-
-#convert '\\e' sequencies to 'x1b' and '\\' to '\'
-cat $LNGISO.po | sed 's/\\e/\\x1b/g;s/\\\\/\\/g' > $LNG'_filtered.po'
-
-#replace '\n' with ' ' (single space)
-sed -i 's/ \\n/ /g;s/\\n/ /g' $LNG'_filtered.po'
-
-#replace in czech translation
-if [ "$LNG" = "cz" ]; then
- #replace 'Á' with 'A'
- sed -i 's/\xc3\x81/A/g' $LNG'_filtered.po'
- #replace 'á' with 'a'
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace 'Č' with 'C'
- sed -i 's/\xc4\x8c/C/g' $LNG'_filtered.po'
- #replace 'č' with 'c'
- sed -i 's/\xc4\x8d/c/g' $LNG'_filtered.po'
- #replace 'Ď' with 'D'
- sed -i 's/\xc4\x8e/D/g' $LNG'_filtered.po'
- #replace 'ď' with 'd'
- sed -i 's/\xc4\x8f/d/g' $LNG'_filtered.po'
- #replace 'É' with 'E'
- sed -i 's/\xc3\x89/E/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'Ě' with 'E'
- sed -i 's/\xc4\x9a/E/g' $LNG'_filtered.po'
- #replace 'ě' with 'e'
- sed -i 's/\xc4\x9b/e/g' $LNG'_filtered.po'
- #replace 'Í' with 'I'
- sed -i 's/\xc3\x8d/I/g' $LNG'_filtered.po'
- #replace 'í' with 'i'
- sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
- #replace 'Ň' with 'N'
- sed -i 's/\xc5\x87/N/g' $LNG'_filtered.po'
- #replace 'ň' with 'n'
- sed -i 's/\xc5\x88/n/g' $LNG'_filtered.po'
- #replace 'Ó' with 'O'
- sed -i 's/\xc3\x93/O/g' $LNG'_filtered.po'
- #replace 'ó' with 'o'
- sed -i 's/\xc3\xb3/o/g' $LNG'_filtered.po'
- #replace 'Ř' with 'R'
- sed -i 's/\xc5\x98/R/g' $LNG'_filtered.po'
- #replace 'ř' with 'r'
- sed -i 's/\xc5\x99/r/g' $LNG'_filtered.po'
- #replace 'Š' with 'S'
- sed -i 's/\xc5\xa0/S/g' $LNG'_filtered.po'
- #replace 'š' with 's'
- sed -i 's/\xc5\xa1/s/g' $LNG'_filtered.po'
- #replace 'Ť' with 'T'
- sed -i 's/\xc5\xa4/T/g' $LNG'_filtered.po'
- #replace 'ť' with 't'
- sed -i 's/\xc5\xa5/t/g' $LNG'_filtered.po'
- #replace 'Ú' with 'U'
- sed -i 's/\xc3\x9a/U/g' $LNG'_filtered.po'
- #replace 'ú' with 'u'
- sed -i 's/\xc3\xba/u/g' $LNG'_filtered.po'
- #replace 'Ů' with 'U'
- sed -i 's/\xc5\xae/U/g' $LNG'_filtered.po'
- #replace 'ů' with 'u'
- sed -i 's/\xc5\xaf/u/g' $LNG'_filtered.po'
- #replace 'Ý' with 'Y'
- sed -i 's/\xc3\x9d/Y/g' $LNG'_filtered.po'
- #replace 'ý' with 'y'
- sed -i 's/\xc3\xbd/y/g' $LNG'_filtered.po'
- #replace 'Ž' with 'Z'
- sed -i 's/\xc5\xbd/Z/g' $LNG'_filtered.po'
- #replace 'ž' with 'z'
- sed -i 's/\xc5\xbe/z/g' $LNG'_filtered.po'
-fi
-
-#replace in German translation https://en.wikipedia.org/wiki/German_orthography
-#replace in Swedish as well
-if [[ "$LNG" = "de" || "$LNG" = "sv" ]]; then
-#replace UTF-8 'äöüß' to HD44780 A00 'äöüß'
- #replace 'ä' with 'A00 ROM ä'
- sed -i 's/\xc3\xa4/\\xe1/g' $LNG'_filtered.po'
- #replace 'Ä' with 'A00 ROM ä'
- sed -i 's/\xc3\x84/\\xe1/g' $LNG'_filtered.po'
- #replace 'ü' with 'A00 ROM ü'
- sed -i 's/\xc3\xbc/\\xf5/g' $LNG'_filtered.po'
- #replace 'Ü' with 'A00 ROM ü'
- sed -i 's/\xc3\x9c/\\xf5/g' $LNG'_filtered.po'
- #replace 'ö' with 'A00 ROM ö'
- sed -i 's/\xc3\xb6/\\xef/g' $LNG'_filtered.po'
- #replace 'Ö' with 'A00 ROM ö'
- sed -i 's/\xc3\x96/\\xef/g' $LNG'_filtered.po'
- #replace 'ß' with 'A00 ROM ß'
- sed -i 's/\xc3\x9f/\\xe2/g' $LNG'_filtered.po'
-fi
-
-#replace in spain translation
-if [ "$LNG" = "es" ]; then
- #replace 'á' with 'a'
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace '¿' with '?'
- sed -i 's/\xc2\xbf/?/g' $LNG'_filtered.po'
- #replace 'ó' with 'o'
- sed -i 's/\xc3\xb3/o/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'í' with 'i'
- sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
- #replace '!' with '!'
- sed -i 's/\xc2\xa1/!/g' $LNG'_filtered.po'
- #replace 'n~' with 'n'
- sed -i 's/\xc3\xb1/n/g' $LNG'_filtered.po'
-fi
-
-#replace in french translation https://en.wikipedia.org/wiki/French_orthography
-if [ "$LNG" = "fr" ]; then
- #replace 'á' with 'a' (right)
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace 'Á' with 'A' (right)
- sed -i 's/\xc3\x81/A/g' $LNG'_filtered.po'
- #replace 'à' with 'a' (left)
- sed -i 's/\xc3\xa0/a/g' $LNG'_filtered.po'
- #replace 'À' with 'A' (left)
- sed -i 's/\xc3\x80/A/g' $LNG'_filtered.po'
- #replace 'é' with 'e' (right)
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'É' with 'E' (right)
- sed -i 's/\xc3\x89/E/g' $LNG'_filtered.po'
- #replace 'è' with 'e' (left)
- sed -i 's/\xc3\xa8/e/g' $LNG'_filtered.po'
- #replace 'È' with 'E' (left)
- sed -i 's/\xc3\x88/E/g' $LNG'_filtered.po'
-fi
-
-#replace in italian translation
-if [ "$LNG" = "it" ]; then
- #replace 'é' with 'e' (left)
- sed -i 's/\xc3\xa8/e/g' $LNG'_filtered.po'
- #replace 'á' with 'a' (left)
- sed -i 's/\xc3\xa0/a/g' $LNG'_filtered.po'
- #replace 'ó' with 'o' (left)
- sed -i 's/\xc3\xb2/o/g' $LNG'_filtered.po'
- #replace 'ú' with 'u' (left)
- sed -i 's/\xc3\xb9/u/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'É' with 'E' (left)
- sed -i 's/\xc3\x88/E/g' $LNG'_filtered.po'
-fi
-
-#replace in dutch translation according to https://nl.wikipedia.org/wiki/Accenttekens_in_de_Nederlandse_spelling
-if [ "$LNG" = "nl" ]; then
- #replace 'ë' with 'e'
- sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
- #replace 'ï' with 'i'
- sed -i 's/\xc3\xaf/i/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'è' with 'e' (left)
- sed -i 's/\xc3\xa8/e/g' $LNG'_filtered.po'
- #replace 'ö' with 'o' (left)
- sed -i 's/\xc3\xb6/o/g' $LNG'_filtered.po'
- #replace 'ê' with 'e' (left)
- sed -i 's/\xc3\xaa/e/g' $LNG'_filtered.po'
- #replace 'ü' with 'u' (left)
- sed -i 's/\xc3\xbc/u/g' $LNG'_filtered.po'
- #replace 'ç' with 'c' (left)
- sed -i 's/\xc3\xa7/c/g' $LNG'_filtered.po'
- #replace 'á' with 'a' (left)
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace 'à' with 'a' (left)
- sed -i 's/\xc3\xa0/a/g' $LNG'_filtered.po'
- #replace 'ä' with 'a' (left)
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'û' with 'u' (left)
- sed -i 's/\xc3\xbc/u/g' $LNG'_filtered.po'
- #replace 'î' with 'i' (left)
- sed -i 's/\xc3\xae/i/g' $LNG'_filtered.po'
- #replace 'í' with 'i' (left)
- sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
- #replace 'ô' with 'o' (left)
- sed -i 's/\xc3\xb4/o/g' $LNG'_filtered.po'
- #replace 'ú' with 'u' (left)
- sed -i 's/\xc3\xba/u/g' $LNG'_filtered.po'
- #replace 'ñ' with 'n' (left)
- sed -i 's/\xc3\xb1/n/g' $LNG'_filtered.po'
- #replace 'â' with 'a' (left)
- sed -i 's/\xc3\xa2/a/g' $LNG'_filtered.po'
- #replace 'Å' with 'A' (left)
- sed -i 's/\xc3\x85/A/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "sv" ]; then
-#repace 'Å' with 'A'
-sed -i 's/\xc3\x85/A/g' $LNG'_filtered.po'
-#repace 'å' with 'a'
-sed -i 's/\xc3\xA5/a/g' $LNG'_filtered.po'
-fi
-
-#https://en.wikipedia.org/wiki/Norwegian_orthography éèêóòôù ÅåÆæØø
-if [ "$LNG" = "no" ]; then
- #replace UTF-8 'æÆøØ' to HD44780 A00 'äö'
- #repace 'Æ' with 'Ä'
- sed -i 's/\xc3\x86/\\xe1/g' $LNG'_filtered.po'
- #repace 'æ' with 'ä'
- sed -i 's/\xc3\xa6/\\xe1/g' $LNG'_filtered.po'
- #repace 'Ø' with 'Ö'
- sed -i 's/\xc3\x98/\\xef/g' $LNG'_filtered.po'
- #repace 'ø' with 'ö'
- sed -i 's/\xc3\xb8/\\xef/g' $LNG'_filtered.po'
- #replace diacritics
- #repace 'Å' with 'A'
- sed -i 's/\xc3\x85/A/g' $LNG'_filtered.po'
- #repace 'å' with 'a'
- sed -i 's/\xc3\xa5/a/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'è' with 'e'
- sed -i 's/\xc3\xa8/e/g' $LNG'_filtered.po'
- #replace 'ê' with 'e'
- sed -i 's/\xc3\xaa/e/g' $LNG'_filtered.po'
- #replace 'ó' with 'o'
- sed -i 's/\xc3\xb3/o/g' $LNG'_filtered.po'
- #replace 'ò' with 'o'
- sed -i 's/\xc3\xb2/o/g' $LNG'_filtered.po'
- #replace 'ô' with 'o'
- sed -i 's/\xc3\xb4/o/g' $LNG'_filtered.po'
- #replace 'ù' with 'u'
- sed -i 's/\xc3\xb9/u/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "da" ]; then
-#repace 'Å' with 'Aa'
-sed -i 's/\xc3\x85/Aa/g' $LNG'_filtered.po'
-#repace 'å' with 'aa'
-sed -i 's/\xc3\xA5/aa/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "sl" ]; then
- #replace 'ë' with 'e'
- sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
- #replace 'ä' with 'a' (left)
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "hu" ]; then # See https://www.fileformat.info/info/charset/UTF-8/list.htm
- #replace 'Á' with 'A'(acute)
- sed -i 's/\xc3\x81/A/g' $LNG'_filtered.po'
- #replace 'á' with 'a'
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace 'É' with 'E' (acute)
- sed -i 's/\xc3\x89/E/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'Í' with 'I' (acute)
- sed -i 's/\xc3\x8d/I/g' $LNG'_filtered.po'
- #replace 'i̇́' with 'i'
- sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
- #replace 'Ó' with 'O' (acute)
- sed -i 's/\xc3\x93/O/g' $LNG'_filtered.po'
- #replace 'ó' with 'o'
- sed -i 's/\xc3\xb3/o/g' $LNG'_filtered.po'
- #replace 'Ö' with 'O' (diaresis)
- sed -i 's/\xc3\x96/O/g' $LNG'_filtered.po'
- #replace 'ö' with 'o'
- sed -i 's/\xc3\xb6/o/g' $LNG'_filtered.po'
- #replace 'Ő' with 'O' (double acute)
- sed -i 's/\xc5\x90/O/g' $LNG'_filtered.po'
- #replace 'ő' with 'o'
- sed -i 's/\xc5\x91/o/g' $LNG'_filtered.po'
- #replace 'Ú' with 'U' (acute) 
- sed -i 's/\xc3\x9a/U/g' $LNG'_filtered.po'
- #replace 'ú' with 'u'
- sed -i 's/\xc3\xba/u/g' $LNG'_filtered.po'
- #replace 'Ü' with 'U' (diaersis)
- sed -i 's/\xc3\x9c/U/g' $LNG'_filtered.po'
- #replace 'ü' with 'u'
- sed -i 's/\xc3\xbc/u/g' $LNG'_filtered.po'
- #replace 'Ű' with 'U' (double acute)
- sed -i 's/\xc5\xb0/U/g' $LNG'_filtered.po'
- #replace 'ű' with 'u'
- sed -i 's/\xc5\xb1/u/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "lb" ]; then
- #replace 'ë' with 'e'
- sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
- #replace 'ä' with 'a'
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "hr" ]; then
- #replace 'ë' with 'e'
- sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
- #replace 'ä' with 'a'
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
-fi
-
-if [ "$LNG" = "lt" ]; then
- #replace 'ë' with 'e'
- sed -i 's/\xc3\xab/e/g' $LNG'_filtered.po'
- #replace 'ä' with 'a'
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
-fi
-#replace in polish translation
-#if [ "$LNG" = "pl" ]; then
-#fi
-
-#replace in slovak translation
-if [ "$LNG" = "sk" ]; then
- #replace 'Á' with 'A'
- sed -i 's/\xc3\x81/A/g' $LNG'_filtered.po'
- #replace 'á' with 'a'
- sed -i 's/\xc3\xa1/a/g' $LNG'_filtered.po'
- #replace 'ä' with 'a'
- sed -i 's/\xc3\xa4/a/g' $LNG'_filtered.po'
- #replace 'Č' with 'C'
- sed -i 's/\xc4\x8c/C/g' $LNG'_filtered.po'
- #replace 'č' with 'c'
- sed -i 's/\xc4\x8d/c/g' $LNG'_filtered.po'
- #replace 'Ď' with 'D'
- sed -i 's/\xc4\x8e/D/g' $LNG'_filtered.po'
- #replace 'ď' with 'd'
- sed -i 's/\xc4\x8f/d/g' $LNG'_filtered.po'
- #replace 'É' with 'E'
- sed -i 's/\xc3\x89/E/g' $LNG'_filtered.po'
- #replace 'é' with 'e'
- sed -i 's/\xc3\xa9/e/g' $LNG'_filtered.po'
- #replace 'Í' with 'I'
- sed -i 's/\xc3\x8d/I/g' $LNG'_filtered.po'
- #replace 'í' with 'i'
- sed -i 's/\xc3\xad/i/g' $LNG'_filtered.po'
- #replace 'ľ' with 'l'
- sed -i 's/\xc4\xbe/l/g' $LNG'_filtered.po'
- #replace 'Ľ' with 'L'
- sed -i 's/\xc4\xbd/L/g' $LNG'_filtered.po'
- #replace 'Ň' with 'N'
- sed -i 's/\xc5\x87/N/g' $LNG'_filtered.po'
- #replace 'ň' with 'n'
- sed -i 's/\xc5\x88/n/g' $LNG'_filtered.po'
- #replace 'Ó' with 'O'
- sed -i 's/\xc3\x93/O/g' $LNG'_filtered.po'
- #replace 'ó' with 'o'
- sed -i 's/\xc3\xb3/o/g' $LNG'_filtered.po'
- #replace 'ô' with 'o'
- sed -i 's/\xc3\xb4/o/g' $LNG'_filtered.po'
- #replace 'Ô' with 'O'
- sed -i 's/\xc3\x94/O/g' $LNG'_filtered.po'
- #replace 'ŕ' with 'r'
- sed -i 's/\xc5\x95/r/g' $LNG'_filtered.po'
- #replace 'Ŕ' with 'R'
- sed -i 's/\xc5\x94/R/g' $LNG'_filtered.po'
- #replace 'Š' with 'S'
- sed -i 's/\xc5\xa0/S/g' $LNG'_filtered.po'
- #replace 'š' with 's'
- sed -i 's/\xc5\xa1/s/g' $LNG'_filtered.po'
- #replace 'Ť' with 'T'
- sed -i 's/\xc5\xa4/T/g' $LNG'_filtered.po'
- #replace 'ť' with 't'
- sed -i 's/\xc5\xa5/t/g' $LNG'_filtered.po'
- #replace 'Ú' with 'U'
- sed -i 's/\xc3\x9a/U/g' $LNG'_filtered.po'
- #replace 'ú' with 'u'
- sed -i 's/\xc3\xba/u/g' $LNG'_filtered.po'
- #replace 'Ý' with 'Y'
- sed -i 's/\xc3\x9d/Y/g' $LNG'_filtered.po'
- #replace 'ý' with 'y'
- sed -i 's/\xc3\xbd/y/g' $LNG'_filtered.po'
- #replace 'Ž' with 'Z'
- sed -i 's/\xc5\xbd/Z/g' $LNG'_filtered.po'
- #replace 'ž' with 'z'
- sed -i 's/\xc5\xbe/z/g' $LNG'_filtered.po'
-fi
-
-#replace UTF-8 'μ' to HD44780 A00 'μ'
- #replace 'μ' with 'A00 ROM μ'
- sed -i 's/\xce\xbc/\\xe4/g' $LNG'_filtered.po'
-#replace non-break space with space
- sed -i 's/\xc2\xa0/ /g' $LNG'_filtered.po'
-
-#check for nonasci characters except HD44780 ROM A00 'äöüß'
-if grep --color='auto' -P -n '[^\x00-\x7F]' $LNG'_filtered.po' >nonascii.txt; then
- exit
-fi
-
-#join lines with multi-line string constants
-cat $LNG'_filtered.po' | sed ':a;N;$!ba;s/\x22\n\x22//g' > $LNG'_new.po'
-
-#Get counter from po files
-
-CNTTXT=$(grep '^# MSG' -c $LNGISO.po)
-num=1
-echo " selected language=$(tput setaf 2)$LNGISO$(tput sgr 0)" >&2
-#generate new dictionary
-cat ../../lang_en.txt | sed 's/\\/\\\\/g' | while read -r s; do
- /bin/echo -e "$s"
- #echo "s = $s ." >&2
- if [ "${s:0:1}" = "\"" ]; then
- 
-#  /bin/echo -e "$s"
-  s=$(/bin/echo -e "$s")
-  s2=$(grep -F -A1 -B0  "msgid $s" "$LNG"_new.po | tail -n1 | sed 's/^msgstr //')
-  if [ -z "$s2" ]; then
-   echo -ne "  processing $num of $CNTTXT\033[0K\r" >&2
-   echo '"\x00"'
-   num=$((num+1))
-  else
-   echo -ne "  processing $num of $CNTTXT\033[0K\r" >&2
-   echo "$s2"
-   num=$((num+1))
-  fi
-#  echo
- fi
-
-done > lang_en_$LNG.txt
-echo >&2
-echo "$(tput setaf 2)Finished with $LNGISO$(tput sgr 0)" >&2
-#replace two double quotes to "\x00"
-sed -i 's/""/"\\x00"/g' lang_en_$LNG.txt
-#remove CR
-sed -i "s/\r//g" lang_en_$LNG.txt
-
-#check new lang
-python3 ../../lang-check.py $LNG --warn-empty
-#gerenate some output
-python3 ../../lang-check.py $LNG --information >output-layout-$LNG.txt
-grep "msgstr" $LNGISO.po | cut -d '"' -f2 | sort >output-sorted-$LNG.txt
-echo >&2
-echo "$(tput setaf 2)lang-import.sh finished$(tput sgr 0)">&2
-

lang/lang-map.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+from collections import defaultdict
+import argparse
+import elftools.elf.elffile
+import struct
+import sys
+import zlib
+
+from lib.io import warn
+
+
+def warn_sym(name, start, size, msg):
+    warn(f'{name}[{start:x}+{size:x}]: {msg}')
+
+
+def get_lang_symbols(elf, symtab):
+    # fetch language markers
+    pri_start = symtab.get_symbol_by_name("__loc_pri_start")[0].entry.st_value
+    pri_end = symtab.get_symbol_by_name("__loc_pri_end")[0].entry.st_value
+    text_data = elf.get_section_by_name('.text').data()
+
+    # extract translatable symbols
+    syms = []
+    sym_id = 0
+    for sym in sorted(symtab.iter_symbols(), key=lambda x: x.entry.st_value):
+        sym_start = sym.entry.st_value
+        sym_size = sym.entry.st_size
+        sym_end = sym_start + sym_size
+        if sym_start >= pri_start and sym_end < pri_end and sym_size > 0:
+            data = text_data[sym_start:sym_end]
+
+            # perform basic checks on the language section
+            if data[0] != 255 or data[1] != 255:
+                warn_sym(sym.name, sym_start, sym_size, 'invalid location offset')
+            if data[-1] != 0:
+                warn_sym(sym.name, sym_start, sym_size, 'unterminated string')
+
+            syms.append({'start': sym_start,
+                         'size': sym_size,
+                         'name': sym.name,
+                         'id': sym_id,
+                         'data': data[2:-1]})
+            sym_id += 1
+
+    return syms
+
+
+def fw_signature(syms):
+    # any id which is stable when the translatable string do not change would do, so build it out of
+    # the firmware translation symbol table itself
+    data = b''
+    for sym in syms:
+        data += struct.pack("<HHH", sym['start'], sym['size'], sym['id'])
+        data += sym['name'].encode('ascii') + b'\0'
+        data += sym['data'] + b'\0'
+    return zlib.crc32(data)
+
+
+def get_sig_sym(symtab, syms):
+    pri_sym = symtab.get_symbol_by_name('_PRI_LANG_SIGNATURE')[0]
+    pri_sym_data = fw_signature(syms)
+    pri_sym = {'start': pri_sym.entry.st_value,
+               'size': pri_sym.entry.st_size,
+               'name': pri_sym.name,
+               'id': '',
+               'data': pri_sym_data}
+    return pri_sym
+
+
+def patch_binary(path, syms, pri_sym):
+    fw = open(path, "r+b")
+
+    # signature
+    fw.seek(pri_sym['start'])
+    fw.write(struct.pack("<I", pri_sym['data']))
+
+    # string IDs
+    for sym in syms:
+        fw.seek(sym['start'])
+        fw.write(struct.pack("<H", sym['id']))
+
+
+def check_duplicate_data(syms):
+    data_syms = defaultdict(list)
+    for sym in syms:
+        data_syms[sym['data']].append(sym)
+    for data, sym_list in data_syms.items():
+        if len(sym_list) > 1:
+            sym_names = [x['name'] for x in sym_list]
+            warn(f'symbols {sym_names} contain the same data: {data}')
+
+
+def output_map(syms):
+    print('OFFSET\tSIZE\tNAME\tID\tSTRING')
+    for sym in syms:
+        print('{:04x}\t{:04x}\t{}\t{}\t{}'.format(sym['start'], sym['size'], sym['name'], sym['id'], sym['data']))
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('elf', help='Firmware ELF file')
+    ap.add_argument('bin', nargs='?', help='Firmware BIN file')
+    args = ap.parse_args()
+
+    # extract translatable symbols
+    elf = elftools.elf.elffile.ELFFile(open(args.elf, "rb"))
+    symtab = elf.get_section_by_name('.symtab')
+    syms = get_lang_symbols(elf, symtab)
+    pri_sym = get_sig_sym(symtab, syms)
+
+    # do one additional pass to check for symbols containing the same data
+    check_duplicate_data(syms)
+
+    # output the symbol table map
+    output_map(syms + [pri_sym])
+
+    # patch the symbols in the final binary
+    if args.bin is not None:
+        patch_binary(args.bin, syms, pri_sym)
+
+    return 0
+
+if __name__ == '__main__':
+    exit(main())

lang/lang-patchsec.py

@@ -0,0 +1,35 @@
+#!/usr/bin/env python3
+import argparse
+import elftools.elf.elffile
+
+from lib.io import fatal
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('elf', help='Firmware ELF file')
+    ap.add_argument('cat', help='Binary language catalog file')
+    ap.add_argument('bin', help='Firmware BIN file')
+    args = ap.parse_args()
+
+    # get the language table position
+    elf = elftools.elf.elffile.ELFFile(open(args.elf, "rb"))
+    symtab = elf.get_section_by_name('.symtab')
+    lang_table_sym = symtab.get_symbol_by_name('_SEC_LANG')[0]
+    lang_table_start = lang_table_sym.entry.st_value
+    lang_table_size = lang_table_sym.entry.st_size
+
+    # read the binary catalog
+    cat = open(args.cat, "rb").read()
+    if len(cat) > lang_table_size:
+        fatal("language catalog too large")
+
+    # patch the symbol
+    with open(args.bin, "r+b") as fw:
+        fw.seek(lang_table_start)
+        fw.write(cat)
+
+    return 0
+
+if __name__ == '__main__':
+    exit(main())

lang/lib/charset.py

@@ -0,0 +1,59 @@
+# Mapping from LCD source encoding to unicode characters
+CUSTOM_CHARS = {
+    '\x04': '🔃',
+    '\xe4': 'µ',
+    '\xdf': '°',
+    '\xe1': 'ä',
+    '\xe4': 'μ',
+    '\xef': 'ö',
+    '\xf5': 'ü',
+}
+
+# Charaters to be remapped prior to source-encoding transformation
+# This transformation is applied to the translation prior to being converted to the final encoding,
+# and maps UTF8 to UTF8. It replaces unavailable symbols in the translation to a close
+# representation in the source encoding.
+TRANS_CHARS = {
+    'Ä': 'ä',
+    'Å': 'A',
+    'Ö': 'ö',
+    'Ü': 'ü',
+    'å': 'a',
+    'æ': 'ä',
+    'ø': 'ö',
+    'ß': 'ss',
+}
+
+
+def _character_check(buf, valid_chars):
+    for c in buf:
+        if (not c.isascii() or not c.isprintable()) and c not in valid_chars:
+            return c
+    return None
+
+def source_check(buf):
+    valid_chars = set(CUSTOM_CHARS.values())
+    valid_chars.add('\n')
+    return _character_check(buf, valid_chars)
+
+def translation_check(buf):
+    valid_chars = set(CUSTOM_CHARS.keys())
+    valid_chars.add('\n')
+    return _character_check(buf, valid_chars)
+
+
+def source_to_unicode(buf):
+    for src, dst in CUSTOM_CHARS.items():
+        buf = buf.replace(src, dst)
+    return buf
+
+def trans_replace(buf):
+    for src, dst in TRANS_CHARS.items():
+        buf = buf.replace(src, dst)
+    return buf
+
+def unicode_to_source(buf):
+    buf = trans_replace(buf)
+    for dst, src in CUSTOM_CHARS.items():
+        buf = buf.replace(src, dst)
+    return buf

lang/lib/io.py

@@ -0,0 +1,35 @@
+import os
+import sys
+import ast
+
+def info(msg):
+    print(os.path.basename(sys.argv[0]) + ": " + msg)
+
+def warn(msg):
+    print(os.path.basename(sys.argv[0]) + ": " + msg, file=sys.stderr)
+
+def fatal(msg):
+    warn(msg)
+    exit(1)
+
+
+def load_map(path):
+    fd = open(path, "r")
+
+    # check the header
+    if fd.readline() != 'OFFSET\tSIZE\tNAME\tID\tSTRING\n':
+        fatal("invalid map file")
+
+    # parse symbols
+    syms = []
+    for line in fd:
+        line = line.rstrip('\n')
+        offset, size, name, tr_id, data = line.split('\t', 4)
+        data = ast.literal_eval(data)
+        tr_id = int(tr_id) if len(tr_id) else None
+        syms.append({'offset': int(offset, 16),
+                     'size': int(size, 16),
+                     'id': tr_id,
+                     'name': name,
+                     'data': data})
+    return syms

lang/progmem.sh

@@ -1,129 +0,0 @@
-#!/bin/bash
-#
-# Version 1.0.1 Build 12
-#
-# progmem.sh - multi-language support script
-#  Examine content of progmem sections (default is progmem1).
-#
-# Input files:
-#  $OUTDIR/Firmware.ino.elf
-#  $OUTDIR/sketch/*.o (all object files)
-#
-# Output files:
-#  text.sym     - formated symbol listing of section '.text'
-#  $PROGMEM.sym - formated symbol listing of section '.progmemX'
-#  $PROGMEM.lss - disassembly listing file
-#  $PROGMEM.hex - variables - hex
-#  $PROGMEM.chr - variables - char escape
-#  $PROGMEM.var - variables - strings
-#  $PROGMEM.txt - text data only (not used)
-#
-#############################################################################
-# Change log:
-# 31 May  2018, XPila,     Initial
-#  9 June 2020, 3d-gussner, Added version and Change log
-#  9 June 2020, 3d-gussner, colored output
-#  2 Apr. 2021, 3d-gussner, Use `git rev-list --count HEAD progmem.sh`
-#                           to get Build Nr
-#############################################################################
-#
-# Config:
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-if [ -z "$OUTDIR" ]; then echo "$(tput setaf 1)variable OUTDIR not set!$(tput sgr0)" >&2; exit 1; fi
-if [ -z "$OBJDIR" ]; then echo "$(tput setaf 1)variable OBJDIR not set!$(tput sgr0)" >&2; exit 1; fi
-if [ -z "$INOELF" ]; then echo "$(tput setaf 1)variable INOELF not set!$(tput sgr0)" >&2; exit 1; fi
-if [ -z "$OBJDUMP" ]; then echo "$(tput setaf 1)variable OBJDUMP not set!$(tput sgr0)" >&2; exit 1; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
-#
-# Program memory used
-PROGMEM=progmem$1
-if [ -z "$1" ]; then PROGMEM=progmem1; fi
-#
-# Description of process:
-#  0. check input files
-#  1. remove output files
-#  2. list symbol table of section '.text' from output elf file to text.sym (sorted by address)
-#  3. calculate start and stop address of section '.$PROGMEM'
-#  4. extract string data from elf file to $PROGMEM.hex
-#  5. prepare string data for character check and conversion (output to $PROGMEM.chr)
-#  6. perform character check and conversion (output to $PROGMEM.var and $PROGMEM.txt)
-#
-
-echo "$(tput setaf 2)progmem.sh started$(tput sgr0)" >&2
-
-# (0)
-echo " progmem.sh (0) - checking input files" >&2
-if [ ! -e $OUTDIR ]; then echo "progmem.sh - file $(tput setaf 2)"$INOELF"$(tput sgr 0) not found!" >&2; exit 1; fi
-
-# (1)
-echo " progmem.sh (1) - removing output files" >&2
-#remove output files if exists
-if [ -e text.sym ]; then rm text.sym; fi
-if [ -e $PROGMEM.sym ]; then rm $PROGMEM.sym; fi
-if [ -e $PROGMEM.lss ]; then rm $PROGMEM.lss; fi
-if [ -e $PROGMEM.hex ]; then rm $PROGMEM.hex; fi
-if [ -e $PROGMEM.chr ]; then rm $PROGMEM.chr; fi
-if [ -e $PROGMEM.var ]; then rm $PROGMEM.var; fi
-if [ -e $PROGMEM.txt ]; then rm $PROGMEM.txt; fi
-
-# (2)
-echo " progmem.sh (2) - listing symbol table of section '.text'" >&2
-#list symbols from section '.text' into file text.sym (only address, size and name)
-#$OBJDUMP -t -j ".text" $INOELF | sort > text.sym
-$OBJDUMP -t -j ".text" $INOELF | tail -n +5 | grep -E "^[0-9a-f]{8} [gl]     [O ]" | cut -c1-9,28-36,37- | sed "/^$/d" | sort > text.sym
-
-# (3)
-echo " progmem.sh (3) - calculating start and end address" >&2
-#calculate start addres of section ".$PROGMEM"
-PROGMEM_BEG=$(cat text.sym | grep "__loc_pri_start" | while read offs size name; do echo "0x"$offs; done)
-#calculate stop addres of section ".$PROGMEM"
-PROGMEM_END=$(cat text.sym | grep "__loc_pri_end" | while read offs size name; do echo "0x"$offs; done)
-echo "  START address = "$PROGMEM_BEG >&2
-echo "  STOP  address = "$PROGMEM_END >&2
-
-# (4)
-echo " progmem.sh (4) - extracting string data from elf" >&2
-#dump $PROGMEM data in hex format, cut disassembly (keep hex data only)
-$OBJDUMP -D -j ".text" -w -z --start-address=$PROGMEM_BEG --stop-address=$PROGMEM_END $INOELF |\
- tail -n +7 | sed "s/ \t.*$//" > $PROGMEM.lss
-#convert $PROGMEM.lss to $PROGMEM.hex:
-# replace empty lines with '|' (variables separated by empty lines)
-# remove address from multiline variables (keep address at first variable line only)
-# remove '<' and '>:', remove whitespace at end of lines
-# remove line-endings, replace separator with '\n' (join hex data lines - each line will contain single variable)
-cat $PROGMEM.lss | sed -E 's/^$/|/;s/^    ....:\t//;s/[ ]*$/ /' | tr -d '\n' | tr '|' '\n' |\
- sed "s/^ //;s/<//;s/>:/ /;s/00 [1-9a-f][1-9a-f] $/00 /; s/ $//" > $PROGMEM.hex
-
-# (5)
-echo " progmem.sh (5) - preparing string data" >&2
-#convert $PROGMEM.hex to $PROGMEM.chr (prepare string data for character check and conversion) 
-# replace first space with tab
-# replace second and third space with tab and space
-# replace all remaining spaces with '\x'
-# replace all tabs with spaces
-cat $PROGMEM.hex | sed 's/ /\t/;s/  /\t /;s/ /\\x/g;s/\t/ /g' > $PROGMEM.chr
-
-# (6)
-#convert $PROGMEM.chr to $PROGMEM.var (convert data to text)
-echo " progmem.sh (6) - converting string data" >&2
-(\
-echo "/bin\/echo -e \\"; \
-cat $PROGMEM.chr | \
- sed 's/ \\xff\\xff/ /;' | \
- sed 's/\\x22/\\\\\\x22/g;' | \
- sed 's/\\x1b/\\\\\\x1b/g;' | \
- sed 's/\\x01/\\\\\\x01/g;' | \
- sed 's/\\xf8/\\\\\\xf8/g;' | \
- sed 's/\\x0a/\\\\\\x0a/g;' | \
- sed 's/\\x04/\\\\\\x04/g;' | \
- sed 's/\\xe4/\\\\\\xe4/g;' | \
- sed 's/\\n/\\\\\\0a/g;' | \
- sed 's/\\x00$/\n/;s/^/\"/;s/$/\"\\/'; \
-) | sh > $PROGMEM.var
-
-#this step can be omitted because .txt file is not used
-cat $PROGMEM.var | sed 's/\r/\n/g' | sed -E 's/^[0-9a-f]{8} [^ ]* //' >$PROGMEM.txt
-
-echo "$(tput setaf 2)progmem.sh finished$(tput sgr0)" >&2
-
-exit 0

lang/readme.txt

@@ -1,48 +0,0 @@
-Nova podpora vice jazyku ve firmware
-
-
-Zmeny oproti stavajicimu frameworku:
-1. Deklarace lokalizovanych textu primo v kodu, neni nutne udrzovat tabulky.
-2. Zatim dvoj jazycna verze (en_cz, en_de atd). Moznost rozsirit na vicejazycnou (en_cz_de - pro MK2).
-3. Moznost vyberu druheho jazyka ulozeneho v SPI flash (nebude zabirat misto v interni flash, pouze MK3).
-5. Bash postbuild proces namisto perloveho skriptu + nastroje na spravu slovniku.
-
-Popis:
-Novy framework je trochu podobny jako znamy i18n20, ale sity na miru pro AVR atmega s ohledem na maximalni jednoduchost a usporu interni flashe.
-Stringy ktere maji byt prelozene se deklaruji pomoci specialnich maker, zbytek obstara postbuild.
-Vsechny lokalizovane texty se nachazi ve specialni sekci, v pripade AVR musi byt stringy umisteny v dolnich 64kB flash - tzv 'progmem'.
-Po zbuildovani arduinem bude fungovat pouze anglictina, je treba spustit postbuild ktery na zaklade slovniku doplni data sekundarniho jazka a vytvori modifikovany hexfile.
-Jedina data ktera je treba udrzovat jsou slovniky pro preklad. Jsou to textove soubory kde je vzdy sparovan anglicky text s prelozenym textem.
-Kazdy text ve slovniku je jeden radek, muze obsahovat specialni znaky v hexadecimalni podobe (e.g. '\x0a'). Nasledujici radek je vzdy prelozeny text.
-Tento jednoduchy format je zvolen proto aby bylo mozno slovniky a proces prekladu spravovat jen pomoci gitu a nekolika skriptu.
-
-Pokud pridame nebo zmenime nejaky text v kodu, zmeni se po zbuildovani a spusteni nastroje 'update.sh' soubor lang_en_code.txt.
-To je generovany soubor ktery obsahuje vsechny lokalizovane texty pouzite v kodu setridene podle abecedy.
-V gitu uvidime zmenu kterou rucne preneseme do slovniku lang_en_xx.txt, zaroven vytvorime pozadavek na preklad ci korekturu pozadovaneho textu.
-Pokud pridame nebo zmenime nejaky text ve slovnikach, zmeni se po spusteni nastroje 'update.sh' soubor lang_en_dict.txt.
-Ten obsahuje vsechny lokalizovane texty ze slovniku (v anglictine), respektive mnozinu jejich sjednoceni.
-V idealnim pripade by soubory lang_en_code.txt a lang_en_dict.txt mely byt totozne.
-Pokud se zmeni slovnik, je treba znovu vygenerovat binarni soubory lang_en_xx.bin.
-
-
-Pouziti v kodu, priklady:
-
-1. deklarace lokalizovaneho textu v kodu - makro '_i':
-puts_P(_i("Kill all humans!")); //v cz vypise "Zabit vsechny lidi!"
-
-2. deklarace lokalizovaneho textu jako globalni konstanty - makro 'PROGMEM_I1' a 'ISTR':
-const char MSG_PREHEAT[] PROGMEM_I1 = ISTR("Preheat"); //deklarace
-puts_P(get_translation(MSG_PREHEAT)); //v cz vypise "Predehrev"
-
-3. fukce get_translation - zkratka makro '_T':
-puts_P(_T(MSG_PREHEAT)); //v cz vypise "Predehrev"
-
-4. deklarace lokalizovaneho textu jako lokalni promenne - makro '_I':
-const char* text = preheat?_I("Preheat"):_I("Cooldown");
-puts_P(_T(text)); //v cz vypise "Predehrev" nebo "Zchlazeni"
-
-5. deklarace nelokalizovaneho textu - makro 'PROGMEM_N1' a '_n' nebo '_N':
-const char MSG_MK3[] PROGMEM_N1 = "MK3"; //deklarace
-const char* text = _n("MK3");
- nebo
-const char* text = _N("MK3");

lang/textaddr.sh

@@ -1,78 +0,0 @@
-#!/bin/sh
-#
-# Version 1.0.1 Build 7
-#
-# textaddr.sh - multi-language support script
-#  Compile progmem1.var and lang_en.txt files to textaddr.txt file (mapping of progmem addreses to text idenifiers)
-#
-# Input files:
-#  progmem1.var
-#  lang_en.txt
-#
-# Output files:
-#  textaddr.txt
-#
-#
-# Dscription of process:
-#  check if input files exists
-#  create sorted list of strings from progmem1.var and lang_en.txt
-#  lines from progmem1.var will contain addres (8 chars) and english text
-#  lines from lang_en.txt will contain linenumber and english text
-#  after sort this will generate pairs of lines (line from progmem1 first)
-#  result of sort is compiled with simple script and stored into file textaddr.txt
-#
-#############################################################################
-# Change log:
-# 30 May  2018, XPila,     Initial
-#  9 June 2020, 3d-gussner, Added version and Change log
-#  9 June 2020, 3d-gussner, colored output
-#  2 Apr. 2021, 3d-gussner, Use `git rev-list --count HEAD textaddr.sh`
-#                           to get Build Nr
-#############################################################################
-
-echo "$(tput setaf 2)textaddr.sh started$(tput sgr0)" >&2
-
-if [ ! -e progmem1.var ]; then echo "$(tput setaf 1)textaddr.sh - file progmem1.var not found!$(tput sgr0)" >&2; exit 1; fi 
-if [ ! -e lang_en.txt ]; then echo "$(tput setaf 1)textaddr.sh - file lang_en.txt not found!$(tput sgr0)" >&2; exit 1; fi 
-addr=''
-text=''
-(cat progmem1.var | sed -E "s/^([^ ]*) ([^ ]*) (.*)/\1 \"\3\"/";\
- cat lang_en.txt | sed "/^$/d;/^#/d" | sed = | sed '{N;s/\n/ /}') |\
- sort -k2 |\
- sed "s/\\\/\\\\\\\/g" | while read num txt; do
- if [ ${#num} -eq 8 ]; then
-  if [ -z "$addr" ]; then
-   addr=$num
-  else
-   if [ "$text" = "$txt" ]; then
-    addr="$addr $num"
-   else
-    echo "ADDR NF $addr $text"
-    addr=$num
-   fi
-  fi
-  text=$txt   
- else
-  if [ -z "$addr" ]; then
-   if ! [ -z "$num" ]; then echo "TEXT NF $num $txt"; fi
-  else
-   if [ "$text" = "$txt" ]; then
-    if [ ${#addr} -eq 8 ]; then
-     echo "ADDR OK $addr $num"
-    else
-     echo "$addr" | sed "s/ /\n/g" | while read ad; do
-      echo "ADDR OK $ad $num"
-     done
-    fi
-    addr=''
-    text=''
-   else
-    if ! [ -z "$num" ]; then echo "TEXT NF $num $txt"; fi
-   fi
-  fi
- fi
-done > textaddr.txt
-
-echo "$(tput setaf 2)textaddr.sh finished$(tput sgr0)" >&2
-
-exit 0

lang/translations.md

@@ -1,251 +0,0 @@
-# Translations
-
-## Workflow
-
-- Build firmware
-  - using `build.sh`
-  - using `PF-build.sh` with a parameter `-c 1` to keep lang build files after compiling
-- change to `lang` folder
-- check if lang scripts being able to run with `config.sh`
-  - if you get `Arduino main folder: NG` message change in `config.sh` `export ARDUINO=C:/arduino-1.8.5` to `export ARDUINO=<Path to your Arduino IDE folder>`
-    -example: `export ARDUINO=D:/Github/Prusa-Firmware/PF-build-env-1.0.6/windows-64`
-- run `lang-build.sh en` to create english `lang_en.tmp`, `lang_en.dat` and `lang_en.bin` files
-- change in `fw-build.sh` `IGNORE_MISSING_TEXT=1` to `IGNORE_MISSING_TEXT=0` so it stops with error and generates `not_used<variant>.txt` and `not_tran<variant>.txt`
-- run modified `fw-build.sh`
-  - `not_tran<variant>.txt` should be reviewed and added as these are potential missing translations
-    - copy `not_tran<variant>.txt` as `lang_add.txt` 
-	  - check if there are things you don't want to translate or must be modified
-	  - als check that the strings do not start with `spaces` as the scripts doesn't handle these well at this moment.
-	  - run `lang-add.sh lang_add.txt` to add the missing translations to `lang_en.txt` and `lang_en_??.txt`
-  - `not_used<variant>.txt` should only contain messages that aren't used in this variant like MK2.5/S vs MK3/S
-- run `fw-clean.sh` to cleanup firmware related files
-- delete `not_used<variant>.txt` and `not_tran<variant>.txt`
-- run `lang-clean.sh` to cleanup language related files
-- run `lang-export.sh all` to create PO files for translation these are stored in `/lang/po` folder
-  - Send them to translators and reviewers or
-  - copy these to `/lang/po/new` and 
-  - translate these with POEdit the newly added messages
-    - easiest way is to choose `Validate`in POEdit as it shows you `errors` and the `missing translations` / most likely the newly added at the top.
-- The new translated files are expected in `/lang/po/new` folder so store the received files these
-- run `lang-import.sh <language code (iso639-1)>` for each newly translated language
-  - script improvement to import "all" and other things would be great.
-- Double check if something is missing or faulty
-  - run `PF-build.sh -v all -n 1 -c 1` to compile for all variants files
-  - check if there are still some messages in `not_tran<variant>.txt` that need attention
-- After approval
-  - run `fw-clean.sh` to cleanup firmware related files
-  - run `lang-clean.sh` to cleanup language related files
-  - change in `fw-build.sh` back to `IGNORE_MISSING_TEXT=1`
-  - build your firmware with `build.sh`, `PF-build.sh` or how you normally do it.
-  - Check/Test firmware on printer
-
-## Code / usage
-There are 2 modes of operation. If `LANG_MODE==0`, only one language is being used (the default compilation approach from plain Arduino IDE).
-The reset of this explanation is devoted to `LANG_MODE==1`:
-
-`language.h`:
-```C++
-// section .loc_sec (originally .progmem0) will be used for localized translated strings
-#define PROGMEM_I2 __attribute__((section(".loc_sec")))
-// section .loc_pri (originally .progmem1) will be used for localized strings in english
-#define PROGMEM_I1 __attribute__((section(".loc_pri")))
-// section .noloc (originally progmem2) will be used for not localized strings in english
-#define PROGMEM_N1 __attribute__((section(".noloc")))
-#define _I(s) (__extension__({static const char __c[] PROGMEM_I1 = "\xff\xff" s; &__c[0];}))
-#define ISTR(s) "\xff\xff" s
-#define _i(s) lang_get_translation(_I(s))
-#define _T(s) lang_get_translation(s)
-```
-That explains the macros:
-- `_i` expands into `lang_get_translation((__extension__({static const char __c[] PROGMEM_I1 = "\xff\xff" s; &__c[0];})))` . Note the two 0xff's in the beginning of the string. `_i` allows for declaring a string directly in-place of C++ code, no string table is used. The downside of this approach is obvious - the compiler is not able/willing to merge duplicate strings into one.
-- `_T` expands into `lang_get_translation(s)` without the two 0xff's at the beginning. Must be used in conjunction with MSG tables in `messages.h`. Allows to declare a string only once and use many times.
-- `_N` means not-translated. These strings reside in a different segment of memory.
-
-The two 0xff's are somehow magically replaced by real string ID's where the translations are available (still don't know where).
-```C++
-const char* lang_get_translation(const char* s){
-	if (lang_selected == 0) return s + 2; //primary language selected, return orig. str.
-	if (lang_table == 0) return s + 2; //sec. lang table not found, return orig. str.
-	uint16_t ui = pgm_read_word(((uint16_t*)s)); //read string id
-	if (ui == 0xffff) return s + 2; //translation not found, return orig. str.
-	ui = pgm_read_word(((uint16_t*)(((char*)lang_table + 16 + ui*2)))); //read relative offset
-	if (pgm_read_byte(((uint8_t*)((char*)lang_table + ui))) == 0) //read first character
-		return s + 2;//zero length string == not translated, return orig. str.
-	return (const char*)((char*)lang_table + ui); //return calculated pointer
-}
-```
-
-## Files
-
-### `lang_en.txt`
-```
-#MSG_CRASH_DET_ONLY_IN_NORMAL c=20 r=4
-"Crash detection can\x0abe turned on only in\x0aNormal mode"
-```
-
-### `lang_en_*.txt`
-```
-#MSG_CRASH_DET_ONLY_IN_NORMAL c=20 r=4
-"Crash detection can\x0abe turned on only in\x0aNormal mode"
-"Crash detekce muze\x0abyt zapnuta pouze v\x0aNormal modu"
-```
-1. a comment - usually a MSG define with number of characters (c) and rows (r)
-2. English text
-3. translated text
-
-### `not_tran.txt`
-A simple list of strings that are not translated yet.
-
-### `not_used.txt`
-A list os strings not currently used in this variant of the firmware or are obsolete.
-Example: There are MK2.5 specific messages that aren't used when you compile a MK3 variant and vice versa. So be careful and double check the code if this message is obsolete or just not used due to the chosen variant.
-
-## Scripts
-
-### `config.sh`
-- Checks setup and sets auxiliary env vars used in many other scripts. 
-- Looks for env var `ARDUINO`. If not found/empty, a default `C:/arduino-1.8.5` is used.
-- Sets env var `CONFIG_OK=1` when all good, otherwise sets `CONFIG_OK=0`
-
-### `fw-build.sh`
-Joins firmware HEX and language binaries into one file.
-
-### `fw-clean.sh`
-
-### `lang-add.sh`
-Adds new messages into the dictionary regardless of whether there have been any older versions.
-
-### `lang-build.sh`
-Generates lang_xx.bin (language binary files) for the whole firmware build.
-
-Arguments:
-- `$1` : language code (`en`, `cz`, `de`, `es`, `fr`, `it`, `pl`) or `all`
-- empty/no arguments defaults to `all`
-
-Input: `lang_en.txt` or `lang_en_xx.txt`
-
-Output: `lang_xx.bin`
-
-Temporary files: `lang_xx.tmp` and `lang_xx.dat`
-
-Description of the process:
-The script first runs `lang-check.py $1` and removes empty lines and comments (and non-translated texts) into `lang_$1.tmp`.
-The tmp file now contains all translated texts (some of them empty, i.e. "").
-The tmp file is then transformed into `lang_$1.dat`, which is a simple dump of all texts together, each terminated with a `\x00`.
-Format of the `bin` file:
-- 00-01: `A5 5A`
-- 02-03: `B4 4B`
-- 04-05: 2B size
-- 06-07: 2B number of strings
-- 08-09: 2B checksum
-- 0A-0B: 2B lang code hex data: basically `en` converted into `ne`, i.e. characters swapped. Only `cz` is changed into `sc` (old `cs` ISO code).
-- 0C-0D: 2B signature low
-- 0E-0F: 2B signature high
-- 10-(10 + 2*number of strings): table of string offsets from the beginning of this file
-- after the table there are the strings themselves, each terminated with `\x00`
-
-The signature is composed of 2B number of strings and 2B checksum in lang_en.bin. Signature in lang_en.bin is zero.
-
-### `lang-check.sh` and `lang-check.py`
-Both do the same, only lang-check.py is newer, i.e. lang-check.sh is not used anymore.
-lang-check.py makes a binary comparison between what's in the dictionary and what's in the binary.
-
-### `lang-clean.sh`
-Removes all language output files from lang folder. That means deleting:
--  if [ "$1" = "en" ]; then
-  rm_if_exists lang_$1.tmp
- else
-  rm_if_exists lang_$1.tmp
-  rm_if_exists lang_en_$1.tmp
-  rm_if_exists lang_en_$1.dif
-  rm_if_exists lang_$1.ofs
-  rm_if_exists lang_$1.txt
- fi
- rm_if_exists lang_$1_check.dif
- rm_if_exists lang_$1.bin
- rm_if_exists lang_$1.dat
- rm_if_exists lang_$1_1.tmp
- rm_if_exists lang_$1_2.tmp
-
-### `lang-export.sh`
-Exports PO (gettext) for external translators.
-
-### `lang-import.sh`
-Import from PO.
-
-Arguments:
-- `$1` : language code (`en`, `cz`, `de`, `es`, `fr`, `it`, `pl`)
-- empty/no arguments quits the script
-
-Input files: `<language code>.po` files like `de.po`, `es.po`, etc.
-
-Input folder: ´/lang/po/new´
-
-Output files:
-
-Output foler: ´/lang/po/new´
-
-Needed improvements to scripts:
- - add `all` argument
- - update `replace in <language> translations` to all known special characters the LCD display with Japanese ROM cannot display
- - move `lang_en_<language code>.txt` to folder `/lang`
- - cleanup `<language code>_filtered.po`, `<language code>_new.po` and `nonasci.txt`
-
-### `progmem.sh`
-
-Examine content of progmem sections (default is progmem1).
-
-Input:
-- $OUTDIR/Firmware.ino.elf
-- $OUTDIR/sketch/*.o (all object files)
-
-Outputs:
-- text.sym     - formatted symbol listing of section '.text'
-- $PROGMEM.sym - formatted symbol listing of section '.progmemX'
-- $PROGMEM.lss - disassembly listing file
-- $PROGMEM.hex - variables - hex
-- $PROGMEM.chr - variables - char escape
-- $PROGMEM.var - variables - strings
-- $PROGMEM.txt - text data only (not used)
-
-Description of process:
-- check input files
-- remove output files
-- list symbol table of section '.text' from output elf file to text.sym (sorted by address)
-- calculate start and stop address of section '.$PROGMEM'
-- dump $PROGMEM data in hex format, cut disassembly (keep hex data only) into $PROGMEM.lss
-- convert $PROGMEM.lss to $PROGMEM.hex:
- - replace empty lines with '|' (variables separated by empty lines)
- - remove address from multiline variables (keep address at first variable line only)
- - remove '<' and '>:', remove whitespace at end of lines
- - remove line-endings, replace separator with '\n' (join hex data lines - each line will contain single variable)
-- convert $PROGMEM.hex to $PROGMEM.chr (prepare string data for character check and conversion)
- - replace first space with tab
- - replace second and third space with tab and space
- - replace all remaining spaces with '\x'
- - replace all tabs with spaces
-- convert $PROGMEM.chr to $PROGMEM.var (convert data to text) - a set of special characters is escaped here including `\x0a`
-
-
-### `textaddr.sh`
-
-Compiles `progmem1.var` and `lang_en.txt` files to `textaddr.txt` file (mapping of progmem addresses to text identifiers).
-
-Description of process:
-- check if input files exists
-- create sorted list of strings from progmem1.var and lang_en.txt
-- lines from progmem1.var will contain address (8 chars) and english text
-- lines from lang_en.txt will contain line number and english text
-- after sort this will generate pairs of lines (line from progmem1 first)
-- result of sort is compiled with simple script and stored into file textaddr.txt
-
-Input:
-- progmem1.var
-- lang_en.txt
-
-Output:
-- textaddr.txt
-
-
-
-update_lang.sh

lang/update-po.sh

@@ -0,0 +1,14 @@
+#!/bin/sh
+FILES=$@
+[ -z "$FILES" ] && FILES=po/*.po
+
+for file in $FILES; do
+    # convert the po file to unix to avoid garbage with msgmerge
+    dos2unix "$file"
+
+    # merge from the template
+    msgmerge -U -s -N "$file" po/Firmware.pot
+
+    # ... and back
+    unix2dos "$file"
+done

lang/update-pot.sh

@@ -0,0 +1,3 @@
+#!/bin/sh
+# Extract language data in the po subdir to keep the relative paths intact
+cd po && ../lang-extract.py --no-missing -s -o Firmware.pot ../../Firmware/[a-zA-Z]*.[ch]*

lang/update_lang.sh

@@ -1,89 +0,0 @@
-#!/bin/sh
-#
-# Version 1.0.1 Build 10
-#
-# update_lang.sh - multi-language support script
-#  Update secondary language in binary file.
-#
-#############################################################################
-# Change log:
-# 17 June 2018, XPila,     Initial
-#  9 June 2020, 3d-gussner, Added version and Change log
-#  9 June 2020, 3d-gussner, colored output
-#  2 Apr. 2021, 3d-gussner, Use `git rev-list --count HEAD update_lang.sh`
-#                           to get Build Nr
-#############################################################################
-#
-# Config:
-if [ -z "$CONFIG_OK" ]; then eval "$(cat config.sh)"; fi
-if [ -z "$OBJCOPY" ]; then echo "$(tput setaf 1)variable OBJCOPY not set!$(tput sgr0)" >&2; exit 1; fi
-if [ -z "$CONFIG_OK" ] | [ $CONFIG_OK -eq 0 ]; then echo "$(tput setaf 1)Config NG!$(tput sgr0)" >&2; exit 1; fi
-#
-# Selected language:
-LNG=$1
-if [ -z "$LNG" ]; then LNG='cz'; fi
-#
-
-finish()
-{
- echo
- if [ "$1" = "0" ]; then
-  echo "$(tput setaf 2)update_lang.sh finished with success$(tput sgr0)" >&2
- else
-  echo "$(tput setaf 1)update_lang.sh finished with errors!$(tput sgr0)" >&2
- fi
- case "$-" in
-  *i*) echo "press enter key" >&2; read ;;
- esac
- exit $1
-}
-
-echo "$(tput setaf 2)update_lang.sh started$(tput sgr0)" >&2
-echo " selected language=$(tput setaf 2)$LNG$(tput sgr0)" >&2
-
-echo -n " checking files..." >&2
-if [ ! -e text.sym ]; then echo "$(tput setaf 1)NG!  file text.sym not found!$(tput sgr0)" >&2; finish 1; fi
-if [ ! -e lang_$LNG.bin ]; then echo "$(tput setaf 1)NG!  file lang_$LNG.bin not found!$(tput sgr0)" >&2; finish 1; fi
-if [ ! -e firmware.bin ]; then echo "$(tput setaf 1)NG!  file firmware.bin not found!$(tput sgr0)" >&2; finish 1; fi
-echo "OK" >&2
-
-echo -n " checking symbols..." >&2
-#find symbol _SEC_LANG in section '.text'
-sec_lang=$(cat text.sym | grep -E "\b_SEC_LANG\b")
-if [ -z "$sec_lang" ]; then echo "$(tput setaf 1)NG!\n  symbol _SEC_LANG not found!$(tput sgr0)" >&2; finish 1; fi
-#find symbol _PRI_LANG_SIGNATURE in section '.text'
-pri_lang=$(cat text.sym | grep -E "\b_PRI_LANG_SIGNATURE\b")
-if [ -z "$pri_lang" ]; then echo "$(tput setaf 1)NG!\n  symbol _PRI_LANG_SIGNATURE not found!$(tput sgr0)" >&2; finish 1; fi
-echo "OK" >&2
-
-echo " calculating vars:" >&2
-#get pri_lang addres
-pri_lang_addr='0x'$(echo $pri_lang | cut -f1 -d' ')
-echo "  pri_lang_addr   =$pri_lang_addr" >&2
-#get addres and size
-sec_lang_addr='0x'$(echo $sec_lang | cut -f1 -d' ')
-sec_lang_size='0x'$(echo $sec_lang | cut -f2 -d' ')
-echo "  sec_lang_addr   =$sec_lang_addr" >&2
-echo "  sec_lang_size   =$sec_lang_size" >&2
-#calculate lang_table_addr (aligned to 256byte page)
-lang_table_addr=$((256*$((($sec_lang_addr + 255) / 256))))
-printf "  lang_table_addr =0x%04x\n" $lang_table_addr >&2
-#calculate lang_table_size
-lang_table_size=$((256*$((($sec_lang_size - ($lang_table_addr - $sec_lang_addr))/256))))
-printf "  lang_table_size =0x%04x (=%d bytes)\n" $lang_table_size $lang_table_size >&2
-
-#get lang_xx.bin file size
-lang_file_size=$(wc -c lang_$LNG.bin | cut -f1 -d' ')
-printf "  lang_file_size  =0x%04x (=%d bytes)\n" $lang_file_size $lang_file_size >&2
-
-if [ $lang_file_size -gt $lang_table_size ]; then echo "$(tput setaf 1)Lanaguage binary file size too big!$(tput sgr0)" >&2; finish 1; fi
-
-echo "updating 'firmware.bin'..." >&2
-
-dd if=lang_$LNG.bin of=firmware.bin bs=1 seek=$lang_table_addr conv=notrunc 2>/dev/null
-
-#convert bin to hex
-echo "converting to hex..." >&2
-$OBJCOPY -I binary -O ihex ./firmware.bin ./firmware_$LNG.hex
-
-finish 0