PII Detection in Low-Resource Languages Using Explainable Deep Learning Techniques

This study aims to respond to the pressing demand for effective identification of personally identifiable information in languages with limited resources, focusing on Luganda as a specific example. The two main computational problems of focus are (a) the limited availability of annotated PII datasets in low-resource languages hinders the effectiveness of model training for PII detection and (b) the "black-box" nature of deep learning models often leads to a lack of transparency, making it difficult to interpret their decision-making processes limiting the trust and adoption of AI systems in privacy-sensitive applications.

Therefore, this research attempts to solve the above challenges by employing state-of-the-art deep learning techniques with attention mechanisms. These mechanisms have shown promise in capturing the nuances of language, which is crucial for accurately identifying PII in texts as attention mechanism helps models focus on the important parts of input, hence achieving higher accuracy on small datasets and addressing the issue of data scarcity in PII detection.

Furthermore, to ensure the explainability and transparency of the models, Explainable AI techniques will be applied to provide insights into the decision-making process of the deep learning models. The goal is to develop a robust and interpretable PII detection system for low-resource languages like Luganda.

Despite the growing concern for privacy and data security, particularly in low-resource languages, it has been evident that detecting Personally Identifiable Information (PII) in these languages is still not straight forward task as they present unique computational challenges. This study focused on the challanges of :-

a) Computational digital resources and tools are scarce, especially the annotated datasets and pretrained models which limits the development of more accurate PII identification and anonymisation models:

b) Furthermore, it was recognized that the importance of transparency in machine learning which has largely limited the adaption and trust in these models as their process of decision making is mostly hidden

The main goal of this research is to explore and implement deep learning techniques for PII detection in the low-resource language of Luganda, with attention mechanisms and applying XAI. Specifically:

• To Implement various deep learning models with attention mechanisms to accurately identify PII in low resource languages texts like Luganda.
  
• To apply Explainable Artificial Intelligence methods to illustrate model decision-making processes in identifying PII in low resource languages texts like Luganda.
  

The significance of this research lies in its potential to enhance privacy protections within digital ecosystems and to ensure compliance with international data protection regulations. Additionally,it contributes to explainable PII detection in the low-resource linguistic contexts by: (1) The implementation deep learning models for PII detection for low-resource languages, specifically focusing on Luganda, using state-of-the-art deep learning techniques with attention mechanisms, (2) The application of Explainable AI methods to provide transparency and interpretability in the decision-making processes of the deep learning models used for PII detection in low resource languages texts like Luganda, (3) Evaluatioon and discussion of the limitations of XAI and attention mechanisms for PII detection in low resourced languages.

XAI aims to make the predictions of complex models understandable to humans, which is essential for trust and accountability. However, evidently, there are disparities in the effectiveness of XAI when applied to low-resourced languages, including the scarcity of annotated data sets, linguistic tools, and pre-trained models for these languages severely hampers the interpretability [15]. That is, the absence of rich linguistic features, which are readily available for well-studied languages, leads to a diminished capacity for XAI models to provide meaningful interpretations in low-resourced linguistic contexts. This lack of nuanced linguistic understanding is a significant barrier, as PII detection often hinges on subtle contextual cues and language-specific idiosyncrasies [15][18].

Attention mechanisms allow models to ’focus’ on specific parts of the input data when making predictions, seemingly offering an improvement in the model's performance. The reliance of attention mechanisms on extensive training data sets, which are scarce for low-resourced languages, is a critical limitation [3].

Additionally, attention mechanisms typically use the hidden state of the recurrent neural network (RNN) to determine which parts of an input sequence are most important. Named entities are then located based on context information around high attention words which is not always the case, and a systematic way to locate the start and end of an entity is desired [12].

Furthermore, attention mechanisms rely heavily on the existence and appearance of named entities in a sentence. However, this assumption is not always true; entities such as dates, times, and alphanumeric IDs can be equally important as a named entity and must be located and classified in order to consider the document as de-identified [25]. In scenarios characterized by sparse data, such as those involving low-resourced languages, attention weights may reflect artifacts of overfitting or other biases rather than true explanatory factors [25].

Despite the challenges inherent in working with low-resourced languages, there have been promising efforts to bridge the gap in most low resourced language NLP tasks that PII detection could leverage. Various approaches have advanced low resourced language NLP including PII detection, including transfer learning adapts models from high-resourced to low-resourced languages, mitigating data scarcity [4], multilingual models to capture cross-lingual representations, aiding PII detection across diverse languages [26], data augmentation techniques, like back-translation, enrich training data, enhancing model robustness [5], semi-supervised and self-supervised learning leverage unlabeled data, reducing annotation needs [9], domain adaptation methods bridge distribution gaps between languages, and improving model generalization, and collaborative efforts share resources, including datasets and models.

This research leverages the” Conrad747/lg-ner” [1] dataset, an annotated corpus comprising text data in the Luganda language. The dataset, rich with annotations tailored for named entity recognition (NER), is instrumental in identifying potential PII embedded within the texts. The dataset provides tokenised text along with corresponding named entity tags. It is comprised of a total of 2,979 samples, divided into training, validation, and test sets with 2085, 358 and 536 samples respectively. Each sample consists of text sequences annotated with Named Entity Recognition (NER) tags with main features of tokens (sequence of tokens representing the text) and ner tags (sequence of labels corresponding to the NER tags for each token). Each example consists of a sequence of tokens, where each token is associated with a label indicating the named entity type (e.g., person names, organizations, locations, etc.).

Figure 2 shows the most common tags in the dataset, the size of each word in the cloud corresponds to its frequency in the dataset. The word cloud correctly shows that words that are very common in Luganda vocabulary like “mu, ku, nnga, ye, nti” occur more frequently in the dataset since these are mostly joining words used in every sentence

Data preprocessing involved several steps to prepare the datasets for model training. Initially, text normalization standardizes the text format across datasets. Following this, tokenization breaks down sentences into individual words or tokens. The preprocessing phase also includes removing irrelevant features, such as non-linguistic symbols, and encoding the data, particularly the NER tags, to be suitable for model training.

For instance, Text sequences were tokenized using the Auto Tokenizer with truncation and padding to ensure uniform length sequences and NER tags were aligned with the tokenized inputs to maintain consistency between tokens and labels as shown in table 1.

The Exploratory Data Analysis aimed to uncover the distribution, patterns and insights within the data, focusing on Distribution of PIIs, Linguistic features, Frequency and types of PII present in the dataset with the results presented in form of tables and charts.

Figure 3 shows the distribution of NER (Named Entity Recognition) tags across different entity types. Each NER tag corresponds to a specific entity type such as person, organization, location, etc. The countplot displays the frequency of each NER tag in the dataset. From the visualization, certain NER tags like a Person's name (NER tag = [1,2,3,4]) occur from frequently than a DATE (NER tag = [8,9,10,11,12]) for example which might affect how the model performs on that particular tag.

As shown in figure 4, which displays the top 10 most common NER tags in the dataset along with their frequencies to identify the most prevalent entity types recognized by the NER model. From the plot, entity types like PERSON and LOCATION occur more frequently in the dataset.

As illustrated in figure 5, it is evident of insights into the complexity and density of entity mentions within sentences, which can influence model design and performance. The plot shows that most sentences have a good density of entity mentions which would contribute to better training of the model.

The scatter plot in figure 6 visualizes the relationship between sentence length and the number of NER tags. Each point represents a sentence, with its x-coordinate being the sentence length and y-coordinate being the number of NER tags. It helps in identifying any patterns or correlations between these two variables. From the plot, longer sentences tend to have a higher density of Ner tags which would imply that the longer the sentence, the more likely it is to find more PII in that sentence.

Figure 7 shows the distribution of unique token lengths with each point on the plot representing the length of a unique token, sorted by token index.

Figure 8, shows the distribution of tokens categorized by their entity types. Each entity type is represented by a different color in the histogram.

Three distinct deep learnig models were implemented i.e. the luganda-ner-v6, deberta-v3-base, and afroxlmr-large-ner-masakhaner.

The Luganda-NER-v6 model is a specialized model designed for Named Entity Recognition (NER) in Luganda, a language spoken in Uganda. Based on the xlm-roberta-base architecture, this model leverages the Transformer's powerful self-attention mechanism to process text. It comprises about 277M parameters that enables the model to capture complex patterns and features specific to Luganda named entities. It was fine-tuned on a dataset specifically annotated for NER tasks, which includes various entity types such as names, locations, and organizations. The primary role of luganda-ner-v6 is to identify and classify named entities within Luganda text, aiding in tasks like information extraction and data analysis. Its success is reflected in its high accuracy and F1 score, demonstrating the model's precision in understanding and categorizing Luganda entities. Was Selected as it has been fine-tuned on a dataset rich in Luganda entities, enabling it to discern and categorize PII with remarkable accuracy.

DeBERTa-v3-Base, is based on disentangled attention mechanism, which separately processes the content and position of each word in a sentence. This base model has 12-layer architecture and 768 hidden size, it contains 86M backbone parameters with a vocabulary containing 250K tokens which introduces 190M parameters in the Embedding layer. It trained on a massive 160GB dataset. The primary role of deberta-v3-base was machine translation. It has shown remarkable success, outperforming previous models like BERT and RoBERTa on a variety of natural language understanding benchmarks. This success is a testament to the model's ability to decode and interpret complex language patterns more effectively.

Afroxlmr-Large-NER-Masakhaner, is a multilingual NER model that addresses the need for language technology in African languages. Fine-tuned on the MasakhaNER dataset, which includes 20 African languages, this model is based on the Transformer architecture boasting about 559M parameters. The large number of parameters gives it ability to capture a wide range of linguistic features across multiple African languages such as 80.5% for Amharic in MasakhaNER 1.0 to 90.5% for Yorùbá in MasakhaNER 2.0, in dentifying entities of dates, locations, organizations, and persons. Its primary role is to facilitate NER tasks across a wide range of African languages, many of which have been historically underrepresented in NLP research. The model's success lies in its ability to accurately identify named entities across these diverse languages, contributing significantly to the inclusivity of language technologies.

Each of the 3 models were trained on the dataset training and evaluated on the validation and testing set. The training was monitored on epochs by model loss and accuracy gain as shown in figure 9.

From figure 9, it is evident that DeBERTa-v3-Base model has the poorest initial training performance but had the highest learning gain, this suggest that further fine tuning would allow it to produce better results. Despite srting high, luganda-ner-v6 performance keeps flactuating suggesting sensitivity and the Afroxlmr-Large-NER-Masakhaner consistently maintaned an improvement and reached an optimal performance at an earlier epoch (7) than the other models. For each model, a the standard performance metrics such as precision, recall, and the F1 score were utilised to evaluate and compare the models. These metrics were calculated for each category of PII identified within the dataset, providing a assessment of the models’ effectiveness. In parallel, XAI techniques were integrated to offer a window into the inner workings of the models. By applying LIME, the the models’ predictions wer interpreted and shone a light on the influential features that underpin the detection of PII. This level of transparency is not merely a byproduct of the methodology but a deliberate effort to foster trust and understanding in AI systems while simultaneously enhancing the adaptability of the models to the unique linguistic nuances present in low resource languages texts like Luganda.

To demonstrate the model's capabilities, below are some of the sample test cases:

case 1: text = ”Ssemakula yategese ekivvulu okutalaaga ebitundu omuli Buddu ne Bulemeezi.”

The model predicts the following entities: PERSON: Ssemakula LOCATION: Buddu, Bulemeezi

case 2: text = ”Katikiro Ssebugwaawo asisinkanye Minisita wa Kampala Minsa Kabanda”

Model Detected PII entities:[’Katikiro’, ’Ssebugwaawo’,’Minisita’, ’Kampala’, ’Minsa’, ’Kabanda’]

which are indeed PII entries.

A shown in figure 10, the afroxlmr-large-ner-masakhaner model has the highest accuracy with 96.3%, followed closely by luganda-ner-v6 at 95.1%, and deberta-v3-base at 93.9%. Precision, crucial for minimizing false positives, favors afroxlmr-large-ner-masakhaner at 86.5%, luganda-ner-v6 at 82.5%, and deberta-v3-base at 76.2%. The F1 score, balancing precision and recall, highlights afroxlmr-large-ner-masakhaner's lead with 85.5%, luganda-ner-v6 at 82.3%, and deberta-v3-base at 75.4%. Similary, recall emphasizes afroxlmr-large-ner-masakhaner's dominance at 84.4%, followed by luganda-ner-v6 at 82.1%, and deberta-v3-base at 74.5%.

In conclusion, while all models perform well, afroxlmr-large-ner-masakhaner consistently excels, showcasing the need for tailored linguistic models in PII detection within diverse language contexts.

Table 2 shows the results of the PII detection in Luganda text across different entity types. The NORP (Nationalities or religious or political groups) category shows the highest precision at 0.946, indicating that when the model predicts an entity as NORP, it is correct 94.6% of the time. However, its recall is lower at 78.4%, suggesting that the model misses some NORP entities present in the text.

The USERID category has a good balance between precision and recall, with scores of 0.812 and 76.5% respectively, leading to an F1 score of 78.8% This indicates a relatively reliable performance in detecting user IDs within the text.

LOCATION entities are detected with a precision of 79.1% and a recall of 83.8%, resulting in an F1 score of 81.4%. This is noteworthy because it suggests the model is quite adept at identifying locations, which often have clear contextual indicators.

The detection of PERSON names has nearly equal precision and recall, both around 78%, showing that the model has a consistent performance in identifying individual names, although there is room for improvement.

ORG(Organizations) detection has a precision of 77.4% and a recall of 71.9%, with an F1 score of 74.5%. This indicates a moderate level of accuracy in identifying organizations, which can be challenging due to the varied ways organizations can be referred to in text.

The DATE category has the lowest performance with equal precision and recall at 5, leading to an F1 score of 0.5. This suggests significant difficulty in detecting dates, which could be due to the complexity of date formats and expressions in Luganda.

Overall, the model achieves an F1 score of 0.789, which is quite robust. However, the varying performance across different categories highlights the challenges in PII detection in low-resource languages. The results suggest that further refinement is needed, particularly in improving recall for NORP and precision for DATE entities. Additionally, the relatively low number of USERID and DATE entities in the dataset may have influenced the model's ability to learn from these examples effectively.

The applied XAI technique is LIME, which provides explanations for individual predictions made by the models by highlighting the words that contributed the most to the model's prediction. Typically, LIME visualisations (i.e. figures 11, 12, 13) were used to present results which contains the model's predictions,features contributions, and the actual prediction for each feature. The height of each bar indicates the probability assigned by the model to the corresponding NER tag label. The contributions associated with each word in the input sentence indicate how much they influence the model's prediction for specific NER tag labels. The actual prediction section shows the input sentence with varying shades representing the importance of each word in the model's decision-making process.Darker shades imply higher importance to the prediction for the corresponding NER tag labels, while lighter shades suggest less influence.

5.3.1 XAI for luganda-ner-v6. As shown in figure 11, Each index corresponds to a specific NER tag as follows, 18: ’I-ORG’, 19: ’L-ORG’, 4: ’U-PERSON’, 5: ’B-NORP’ "other": represents all other NER tag labels not explicitly listed here, such as ’O’ (non-entity) or other entity types not included in the provided list 1. From the feature contribution graphs and actual senstence prediction for luganda-ner-v6 in figure 11, it can be seen that "asisinkanye" has a higher importance to the prediction of the NER tags for the Luganda NER model, followed by words like "Katikiro", "Ssebugwawo", "Minsa", then lesser importance for "Minisita", "wa", "Kampala", and "Kabanda" respectively.

5.3.2 XAI for deberta-v3-base. From figure 12, we can observe that for the indicies of 18, 19, 4, 5, and "other" which correspond to a specific NER tags of 0: ’O’, 1: ’B-PERSON’, 2: ’I-PERSON’, 3: ’L-PERSON’, "other": all other NER tag labels not explicitly listed here, such as ’B-ORG’, ’I-ORG’, ’L-ORG’, ’U-ORG’, etc. It can be observed from feature contribution graphs and actual senstence prediction in figure 12, that all the words in the sentence "Katikiro Ssebugwaawo asisinkanye Minisita wa Kampala Minsa Kabanda" have equal importance to the prediction of the NER tags for this model.

5.3.3 XAI for afroxlmr-ner-masakhaner. As shown from figure 13, for the indices of 19, 18, 2, 3, and "other" which correspond to NER tag as 19: ’L-ORG’, 18: ’I-ORG’, 2: ’I-PERSON’, 3: ’L-PERSON’ "other": all other NER tag labels, the words "Ssebugwawo", "Kampala" have the highest importance to the prediction of the NER tags for the afroxlmr-ner-masakhaner model, followed by "Minista", "Katikiro", "asisinkanye", and "wa", with "Minsa" and "Kabanda" as the least important.

The models of Luganda-NER-v6, DeBERTa-v3-Base, and Afroxlmr-Large-NER-Masakhane, where each selected not arbitrary but as a calculated decision grounded in the unique attributes and proven efficacy of each model.

Luganda-NER-v6 was selected for its specialization in the Luganda language, Its design and training tailored for Named Entity Recognition (NER) within the NER linguistic context of Luganda text.

The DeBERTa-v3-Base was chosen for its innovative disentangled attention mechanism which enables it to process content and positional information distinctly, achieving a nuanced comprehension of context. This is vital for the accurate detection of PII, where the subtleties of language play a significant role. The model's training on an expansive 160GB dataset equips it with an extensive knowledge base.

From the fact that the Afroxlmr-Large-NER-Masakhaner is a multilingual, having been trained across 20 African languages and in a low resource setting. This positions it as an invaluable tool for PII detection in low-resource languages, including Luganda. Its fine-tuning on the MasakhaNER dataset, which is specifically curated for NER tasks across African languages, ensures its adeptness in pinpointing PII. Moreover, its role in fostering inclusivity within language technologies cannot be overstated.

Similarly, the choice of Local Interpretable Model-agnostic Explanations(LIME) as the XAI technique was deliberate based on model-agnostic Explanations, Feature Importance in Low-Resource Settings and visualisation. LIME offer methods to understand feature importance in low resource settings. Furthermore, the technique is model-agnostic hence flexibility to allow the seamless integration of XAI into the deep learning pipeline regardless of the final model architecture.