Package 'TextAnalysisR'

Description

A dataset containing common acronyms used in text processing

Format

A character vector of acronyms

Description

Records the reasoning behind a coding decision so the interpretive step leaves a trail. Memos attach to a single unit or to a category, carry the round they were written in, and travel with the coded units on export.

Usage

add_memo(
  memos = NULL,
  target_type = c("unit", "category"),
  target_id,
  text,
  round = 1,
  stage = NA_character_
)

Arguments

Value

The memo table with one row appended.

See Also

get_memos() to retrieve them; round_summary() for the rounds memos are stamped with.

Description

Cross-tabulates human against model categories over the same documents and reports the adjusted Rand index, which corrects for chance and does not require the two schemes to share labels or count.

Usage

align_categories(human, machine, unassigned = 0)

Arguments

Value

A list with crosstab (human against machine), adjusted_rand, best_match (each human category paired with the machine category it overlaps most, and the share of the human category that pairing covers), and n (documents compared, excluding missing values).

See Also

validate_categories() for supervised confirmation.

Description

Complete document clustering analysis with dimensionality reduction and optional clustering

Usage

analyze_document_clustering(
  feature_matrix,
  method = "UMAP",
  clustering_method = "none",
  ...
)

Arguments

Value

List containing coordinates, clusters, method info, and quality metrics

Description

Analyzes semantic evolution patterns in temporal results

Usage

analyze_semantic_evolution(temporal_results, verbose = FALSE, ...)

Arguments

Value

List containing evolution analysis

Description

Performs sentiment analysis on text data using the syuzhet package. Classification uses the per-token average score with a +/- 0.05 neutral band, so document length does not drive polarity. Lexicon methods score words in isolation; negation, irony, and idioms are not handled. For negation-sensitive text, see analyze_sentiment_llm().

Usage

analyze_sentiment(texts, method = "syuzhet", doc_ids = NULL)

Arguments

Value

A data frame with columns:

document

Document identifier

text

Original text

sentiment_score

Raw summed sentiment score (length-dependent)

n_tokens

Number of whitespace-delimited tokens

avg_sentiment

sentiment_score / n_tokens

sentiment

Classification: "positive", "negative", or "neutral"

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
sentiment_results <- analyze_sentiment(abstracts)
print(sentiment_results)

Description

Analyzes sentiment using Large Language Models (OpenAI or Gemini). Provides nuanced sentiment understanding including sarcasm detection, mixed emotions, and contextual interpretation that lexicon-based methods miss.

Requests use temperature 0, which makes outputs near-deterministic but not fully reproducible across runs or model versions; record the returned model_used field when reporting results.

Usage

analyze_sentiment_llm(
  texts,
  doc_names = NULL,
  provider = c("openai", "gemini", "ollama"),
  model = NULL,
  api_key = NULL,
  batch_size = 5,
  include_explanation = FALSE,
  verbose = TRUE
)

Arguments

Details

LLM-based sentiment analysis offers several advantages over lexicon methods:

• Understands context and nuance

• Detects sarcasm and irony

• Handles mixed emotions

• Works across domains without retraining

Value

A list containing:

document_sentiment

Data frame with document-level sentiment scores

summary_stats

Summary statistics of the analysis

model_used

Model name used for analysis

provider

AI provider used

Examples

if (interactive()) {
  abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:5]

  sentiment_openai <- analyze_sentiment_llm(abstracts, provider = "openai")

  sentiment_gemini <- analyze_sentiment_llm(abstracts, provider = "gemini",
                                             include_explanation = TRUE)
}

Description

Identifies unique items, missing content, and cross-category learning opportunities based on similarity thresholds. Useful for gap analysis in policy documents, topic comparisons, or any cross-category similarity study.

Usage

analyze_similarity_gaps(
  similarity_data,
  ref_var = "ref_id",
  other_var = "other_id",
  similarity_var = "similarity",
  category_var = "other_category",
  ref_label_var = NULL,
  other_label_var = NULL,
  unique_threshold = 0.6,
  cross_policy_min = 0.6,
  cross_policy_max = 0.8
)

Arguments

Value

A list containing:

Examples

articles <- TextAnalysisR::SpecialEduTech[1:6, ]
articles$display_name <- paste0("d", seq_len(nrow(articles)))
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
dimnames(similarity_matrix) <- list(articles$display_name, articles$display_name)
cross_similarities <- extract_cross_category_similarities(
  similarity_matrix  = similarity_matrix,
  docs_data          = articles,
  reference_category = "thesis",
  compare_categories = "journal_article",
  category_var       = "reference_type",
  id_var             = "display_name"
)
gap_analysis <- analyze_similarity_gaps(
  similarity_data = cross_similarities,
  ref_var = "ref_id",
  other_var = "other_id",
  similarity_var = "similarity",
  category_var = "other_category",
  unique_threshold = 0.6
)
print(gap_analysis$summary_stats)

Description

Suggests codes for each unit of text from a supplied codebook using an AI provider (OpenAI or Gemini). Documents are split into sentences, paragraphs, or whole documents before coding, and each unit receives zero or more codes. Output is a set of suggestions for human confirmation, not final codes.

Usage

apply_codes(
  texts,
  codebook,
  unit = c("paragraph", "sentence", "document"),
  max_codes = 3,
  provider = c("auto", "openai", "gemini", "ollama"),
  model = NULL,
  temperature = 0,
  api_key = NULL,
  max_tokens = NULL,
  delay = 1,
  verbose = TRUE,
  seed = NULL
)

Arguments

Value

A tibble with one row per unit-code pair: doc_id, unit_id, start, end (character offsets of the unit within its document), code, confidence (0 to 1, NA when none was returned or the value was off scale), rationale, status ("ok", "none", or "error"), and error. Returns invisible(NULL) when no provider key is available.

See Also

code_retest() for AI stability; code_agreement() for inter-coder reliability; call_llm_api() for the direct provider call.

Description

Rebuilds the grouping from one round of researcher verdicts. Merges are applied first, because a merge changes what the remaining groups are compared against; each split then re-partitions that group alone, choosing the number of parts with the highest cosine silhouette among those whose every part meets min_size.

Usage

apply_ruling(
  groups,
  ruling,
  embeddings = NULL,
  min_size = 10,
  max_parts = 4,
  seed = 2026
)

Arguments

Details

Merges apply before splits. Merge chains resolve to their last target, so "g1" into "g2" and "g2" into "g3" in one round leave every unit of the three groups in "g3". A split verdict on a group that also receives a merge in the same round partitions the merged units, not the group as it stood before the round. A split verdict on a group merged away in the same round has no effect. Units with an NA group pass through unchanged.

Value

A character vector of new group labels, one per unit. Split parts take the parent label plus a letter ("g1a", "g1b"). A group the rule cannot split into parts of at least min_size keeps its label, with a warning.

See Also

group_evidence(), ruling_settled().

Description

Evaluates the stability of embedding-based topic modeling by running multiple models with different random seeds and comparing their results. High stability (high ARI, consistent keywords) indicates stable topic structure in the data.

Usage

assess_embedding_stability(
  texts,
  n_runs = 5,
  embedding_model = "all-MiniLM-L6-v2",
  select_best = TRUE,
  base_seed = 123,
  verbose = TRUE,
  precomputed_embeddings = NULL,
  ...
)

Arguments

Details

Stability is assessed via:

• Adjusted Rand Index (ARI): Measures agreement in topic assignments across runs

• Keyword Jaccard similarity: Measures overlap in top keywords per topic

• Quality variance: Variance in silhouette scores across runs

Value

A list containing:

• stability_metrics: List with mean_ari, sd_ari, mean_jaccard, quality_variance

• best_model: Best model by silhouette score (if select_best = TRUE)

• all_models: List of all fitted models

• is_stable: Logical, TRUE if mean ARI >= 0.6 (considered stable)

• recommendation: Text recommendation based on stability

• kept_index: Positions of the non-blank input texts the models were fitted on

Examples

if (interactive()) {
  texts <- c("Machine learning for image recognition",
             "Deep learning neural networks",
             "Natural language processing models")

  stability <- assess_embedding_stability(
    texts = texts,
    n_runs = 3,
    verbose = TRUE
  )

  # Check if results are stable
  stability$is_stable
  stability$stability_metrics$mean_ari

  # Use the best model
  best_model <- stability$best_model
}

Description

Places documents left unassigned by density-based clustering, by similarity to confirmed category members. Distance is returned so weak placements stay visible, and max_distance leaves distant documents unassigned.

Usage

assign_noise(
  embeddings,
  categories,
  k = 5,
  max_distance = NULL,
  unassigned = 0
)

Arguments

Value

A tibble with index, assigned, and distance, one row per previously unassigned document.

See Also

validate_categories() to test the categories afterwards.

Description

Automatically searches for optimal hyperparameters for embedding-based topic modeling. Evaluates multiple configurations of UMAP and HDBSCAN parameters and returns the best model based on the specified metric. Embeddings are generated once and reused across all configurations for efficiency.

Usage

auto_tune_embedding_topics(
  texts,
  embeddings = NULL,
  embedding_model = "all-MiniLM-L6-v2",
  n_trials = 12,
  metric = "silhouette",
  seed = 123,
  verbose = TRUE
)

Arguments

Details

The function searches over these parameters:

• n_neighbors: UMAP neighborhood size (5, 10, 15, 25)

• min_cluster_size: HDBSCAN minimum cluster size (3, 5, 10)

• cluster_selection_method: "eom" (broader) or "leaf" (finer-grained)

Value

A list containing:

• best_config: Data frame with the optimal hyperparameter configuration

• best_model: The topic model fitted with optimal parameters

• all_results: List of all evaluated configurations with metrics

• n_trials_completed: Number of configurations successfully evaluated

Examples

if (interactive()) {
  texts <- c("Machine learning for image recognition",
             "Deep learning neural networks",
             "Natural language processing models",
             "Computer vision applications")

  tuning_result <- auto_tune_embedding_topics(
    texts = texts,
    n_trials = 6,
    metric = "silhouette",
    verbose = TRUE
  )

  # View best configuration
  tuning_result$best_config

  # Use the best model
  best_model <- tuning_result$best_model
}

Description

Calculates consistency between two sets of assignments

Usage

calculate_assignment_consistency(assignments1, assignments2, ...)

Arguments

Value

List containing consistency metrics

Description

Calculates common clustering evaluation metrics including Silhouette Score, Davies-Bouldin Index, and Calinski-Harabasz Index.

Usage

calculate_clustering_metrics(
  clusters,
  data_matrix,
  dist_matrix = NULL,
  metrics = "all"
)

Arguments

Details

• Silhouette Score: Measures how similar an object is to its own cluster compared to other clusters. Range: -1 to 1, higher is better.

• Davies-Bouldin Index: Average similarity between each cluster and its most similar cluster. Lower values indicate better clustering.

• Calinski-Harabasz Index: Ratio of between-cluster to within-cluster variance. Higher values indicate better-defined clusters.

Value

A named list containing:

silhouette

Silhouette score (-1 to 1, higher is better)

davies_bouldin

Davies-Bouldin index (lower is better)

calinski_harabasz

Calinski-Harabasz index (higher is better)

n_clusters

Number of clusters

cluster_sizes

Table of cluster sizes

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:20]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(abstracts)))
kmeans_result <- stats::kmeans(term_matrix, centers = 2)
metrics <- calculate_clustering_metrics(kmeans_result$cluster, term_matrix)
print(metrics)

Description

Mean pairwise cosine similarity of documents within each topic. This is a document-embedding measure, not word-based coherence (UMass, NPMI, C_v). Every label in topic_assignments is scored, including any outlier label.

Usage

calculate_coherence(embeddings, topic_assignments)

Arguments

Value

A list with coherence_scores, mean_coherence, and topic_coherence (named by topic).

Description

Calculates cosine similarity between two different embedding matrices, useful for comparing documents/topics across different categories or groups.

Usage

calculate_cross_similarity(
  embeddings1,
  embeddings2,
  labels1 = NULL,
  labels2 = NULL,
  normalize = TRUE
)

Arguments

Value

A list containing:

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:6]
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(abstracts)))
similarity_result <- calculate_cross_similarity(
  term_matrix[1:3, ], term_matrix[4:6, ],
  labels1 = paste("Doc", 1:3),
  labels2 = paste("Doc", 4:6)
)
similarity_result$similarity_matrix

Description

Computes quantitative dispersion metrics for terms, measuring how evenly distributed they are across the corpus.

Usage

calculate_dispersion_metrics(tokens_object, terms)

Arguments

Value

Data frame with columns:

• term: The search term

• frequency: Total occurrences

• doc_count: Number of documents containing term

• doc_ratio: Proportion of documents containing term

• juilland_d: Juilland's D dispersion (0-1, higher = more even), computed on within-document proportions

• rosengren_s: Rosengren's S dispersion adjusted for document sizes (1/n_docs to 1, higher = more even)

Description

Calculates stability between the topic keyword sets of two independently fitted models. Topic numbering is arbitrary across fits, so topics are matched one-to-one by greedy assignment on the pairwise keyword Jaccard matrix before averaging matched-pair similarity.

Usage

calculate_keyword_stability(keywords1, keywords2)

Arguments

Value

Mean Jaccard similarity of matched topic pairs (0-1).

Description

Computes lexical dispersion data for specified terms across a corpus. Shows where terms appear within each document, useful for understanding term distribution patterns.

Usage

calculate_lexical_dispersion(
  tokens_object,
  terms,
  scale = c("relative", "absolute")
)

Arguments

Value

Data frame with columns:

• doc_id: Document identifier

• term: The search term

• position: Position in document (relative or absolute)

• doc_length: Total tokens in document

Examples

tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1:5])
dispersion <- calculate_lexical_dispersion(tokens, c("learning", "instruction"))

Description

Compares word usage between categories using the log-odds ratio with a uniform Dirichlet prior, ranked by z-score. Identifies words distinctively used in one category. For the informative-prior weighted log-odds, see calculate_weighted_log_odds().

Usage

calculate_log_odds_ratio(
  dfm_object,
  group_var,
  comparison_mode = c("binary", "one_vs_rest", "pairwise"),
  reference_level = NULL,
  top_n = 10,
  min_count = 5
)

Arguments

Value

Data frame with columns:

• term: The word/feature

• category1: First category in comparison

• category2: Second category in comparison

• count1: Count in category 1

• count2: Count in category 2

• odds1: Smoothed odds in category 1, (count + 1) / (total + V - count - 1)

• odds2: Smoothed odds in category 2

• odds_ratio: Ratio of odds

• log_odds_ratio: Log of odds ratio (positive = more in compared category)

• variance: Approximate variance of the log odds ratio

• z_score: Log odds ratio divided by its standard error

• significant: TRUE when the Benjamini-Hochberg adjusted two-sided p-value of z is below 0.05, adjusted across all comparisons in the run

Terms are ranked by absolute z-score.

Examples

articles <- TextAnalysisR::SpecialEduTech[1:20, ]
corpus <- quanteda::corpus(
  articles$abstract,
  docvars = data.frame(reference_type = articles$reference_type)
)
dfm_object <- quanteda::dfm(quanteda::tokens(corpus))
log_odds <- calculate_log_odds_ratio(dfm_object, "reference_type")

Description

Calculates semantic drift across time periods

Usage

calculate_semantic_drift(temporal_results, ...)

Arguments

Value

List containing drift metrics

Description

Calculates document similarity with fallback methods and diagnostics. Attempts embeddings first, falls back to Jaccard similarity if needed.

Usage

calculate_similarity_robust(
  texts,
  method = "embeddings",
  embedding_model = "all-MiniLM-L6-v2",
  cache_embeddings = TRUE,
  min_word_length = 3,
  doc_names = NULL
)

Arguments

Value

List containing similarity matrix, method used, embeddings, and diagnostics

Examples

if (interactive()) {
  abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:5]
  similarity_result <- calculate_similarity_robust(abstracts)
  print(similarity_result$similarity_matrix)
  print(similarity_result$diagnostics)
}

Description

Calculates multiple readability metrics for texts including Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning FOG index, and others. Optionally includes lexical diversity metrics and sentence statistics.

Usage

calculate_text_readability(
  texts,
  metrics = c("flesch", "flesch_kincaid", "gunning_fog"),
  include_lexical_diversity = TRUE,
  include_sentence_stats = TRUE,
  doc_names = NULL
)

Arguments

Value

A data frame with document names, readability scores, and sentence_flag: TRUE for documents with a mean sentence length over 40 words (a single long sentence included), where missing sentence boundaries (common in PDF lines or unpunctuated text) distort sentence-based formulas. Flagged documents keep their scores, and a warning gives their count.

Examples

data(SpecialEduTech, package = "TextAnalysisR")
texts <- SpecialEduTech$abstract[1:10]
readability <- calculate_text_readability(texts)
print(readability)

Description

Extracts and summarizes topic probabilities (gamma values) from an STM model, returning a formatted data table of mean topic prevalence.

Usage

calculate_topic_probability(stm_model, top_n = 10, verbose = TRUE, ...)

Arguments

Value

A DT::datatable showing topics and their mean gamma (prevalence) values, rounded to 3 decimal places.

Examples

if (interactive()) {
  data <- TextAnalysisR::SpecialEduTech
  united <- unite_cols(data, c("title", "keyword", "abstract"))
  tokens <- prep_texts(united, text_field = "united_texts")
  dfm_obj <- quanteda::dfm(tokens)
  stm_data <- quanteda::convert(dfm_obj, to = "stm")

  topic_model <- stm::stm(
    documents = stm_data$documents,
    vocab = stm_data$vocab,
    K = 10,
    verbose = FALSE
  )

  prob_table <- calculate_topic_probability(topic_model, top_n = 10)
  print(prob_table)
}

Description

Calculates stability of topics across consecutive time periods. Each period is fitted independently, so topics are matched one-to-one on keyword Jaccard similarity before scoring (see calculate_keyword_stability()).

Usage

calculate_topic_stability(temporal_results)

Arguments

Value

Stability metrics: matched-pair stability per consecutive-period transition and their mean.

Description

Computes weighted log odds ratios using the method from Monroe, Colaresi, and Quinn (2008) "Fightin' Words" via the tidylo package. This method weights log odds by variance (z-score) to identify words that reliably distinguish between groups, accounting for sampling variability.

tidylo estimates the informative Dirichlet prior from the pooled corpus counts and compares each group with the pooled corpus, so the z-scores are not directly comparable with a background-corpus prior as in Monroe et al.

Usage

calculate_weighted_log_odds(dfm_object, group_var, top_n = 10, min_count = 5)

Arguments

Value

A data frame with the grouping variable, feature, n, log_odds_weighted (from tidylo::bind_log_odds), and significant (TRUE when the Benjamini-Hochberg adjusted two-sided p-value of the z-score is below 0.05)

References

Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4), 372-403.

Silge, J., & Robinson, D. (2017). Text mining with R: A tidy approach. O'Reilly Media. https://www.tidytextmining.com/

Examples

if (requireNamespace("tidylo", quietly = TRUE)) {
  articles <- TextAnalysisR::SpecialEduTech[1:20, ]
  dfm_object <- quanteda::dfm(quanteda::tokens(articles$abstract))
  quanteda::docvars(dfm_object, "reference_type") <- articles$reference_type
  weighted_odds <- calculate_weighted_log_odds(dfm_object, "reference_type",
                                                top_n = 5)
}

Description

This function analyzes and visualizes word frequencies across a continuous variable.

Usage

calculate_word_frequency(
  dfm_object,
  continuous_variable,
  selected_terms,
  height = 500,
  width = 900
)

Arguments

Details

This function requires a fitted STM model object and a quanteda dfm object. The continuous variable should be a column in the metadata of the dfm object. The selected terms should be a vector of terms to analyze trends for. The required packages are 'htmltools', 'splines', and 'broom' (plus additional ones loaded internally).

Value

A list containing Plotly objects and tables with the results.

See Also

extract_keywords_tfidf() and extract_keywords_keyness() for ranking terms by importance; plot_word_frequency() for the standard frequency-by-doc plot

Examples

mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  word_freq_results <- TextAnalysisR::calculate_word_frequency(
    dfm_object,
    continuous_variable = "year",
    selected_terms = c("calculator", "computer"),
    height = 500,
    width = 900
  )
  print(word_freq_results$plot)
  print(word_freq_results$table)

Description

Calls different LLM providers (OpenAI, Gemini, or a local Ollama server) through a single interface. Automatically routes to the appropriate provider-specific function.

Usage

call_llm_api(
  provider = c("openai", "gemini", "ollama"),
  system_prompt,
  user_prompt,
  model = NULL,
  temperature = 0,
  max_tokens = 150,
  api_key = NULL,
  seed = NULL
)

Arguments

Value

Character string with the model's response. OpenAI responses carry served_model and system_fingerprint attributes.

See Also

sanitize_llm_input() to clean text before prompting; get_best_embeddings() for vector embeddings; run_rag_search() for RAG search (retrieval + generation)

Examples

if (interactive()) {
# Using OpenAI
response <- call_llm_api(
  provider = "openai",
  system_prompt = "You are a helpful assistant.",
  user_prompt = "Generate a topic label",
  api_key = Sys.getenv("OPENAI_API_KEY")
)

# Using Gemini
response <- call_llm_api(
  provider = "gemini",
  system_prompt = "You are a helpful assistant.",
  user_prompt = "Generate a topic label",
  api_key = Sys.getenv("GEMINI_API_KEY")
)
}

Description

Clears the internal cache used for lexical diversity calculations. Call this function to free memory or force fresh calculations.

Usage

clear_lexdiv_cache()

Value

Invisible NULL

Description

Groups documents into topics by clustering transformer embeddings, with per-topic cohesion diagnostics.

Usage

cluster_embedding_topics(
  texts,
  n_topics = 10,
  embedding_model = "all-MiniLM-L6-v2",
  clustering_method = c("kmeans", "hierarchical"),
  min_topic_size = 3,
  seed = 123,
  precomputed_embeddings = NULL
)

Arguments

Value

List with topic assignments and diagnostics. The cohesion values are mean pairwise cosine similarity of document embeddings within each topic cluster (embedding-space compactness), not lexical coherence measures such as C_v or NPMI.

See Also

find_optimal_k() and auto_tune_embedding_topics() for choosing n_topics; fit_embedding_model() for the UMAP and HDBSCAN pipeline that derives the topic count from density instead.

Description

This function performs clustering analysis using various methods, ordered from simple to detailed: k-means (simplest), hierarchical (intermediate), and UMAP+DBSCAN (most detailed).

Usage

cluster_embeddings(
  data_matrix,
  method = "kmeans",
  n_clusters = 0,
  umap_neighbors = 15,
  umap_min_dist = 0.1,
  umap_n_components = 10,
  umap_metric = "cosine",
  dbscan_eps = 0,
  dbscan_min_samples = NULL,
  reduce_outliers = FALSE,
  outlier_strategy = "centroid",
  outlier_threshold = 0,
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A list containing cluster assignments, method used, and quality metrics. All-zero rows (empty documents) are not clustered: they get label 0 and their row indices are returned in empty_documents. For method = "umap_dbscan" with reduce_outliers = TRUE, the list also reports how noise points were reassigned: outlier_strategy ("embeddings" or "centroid"), outlier_metric ("cosine" or "euclidean"), outlier_space ("original embedding space" or "UMAP space"), and outliers_reassigned (count).

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  data_matrix <- as.matrix(dfm_object)

  kmeans_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "kmeans",
    n_clusters = 5
  )
  print(kmeans_result)

  hierarchical_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "hierarchical",
    n_clusters = 5
  )
  print(hierarchical_result)

  umap_dbscan_result <- TextAnalysisR::cluster_embeddings(
    data_matrix,
    method = "umap_dbscan",
    umap_neighbors = 15,
    umap_min_dist = 0.1
  )
  print(umap_dbscan_result)
}

Description

Chance-corrected agreement between two or more coders' code assignments. Reports Krippendorff's alpha and Cohen's/Fleiss' kappa (via irr) plus percent agreement, Gwet's AC1, and PABAK computed inline. AC1 and PABAK stay stable under skewed code prevalence, where kappa collapses (the kappa paradox).

Usage

code_agreement(
  assignments,
  metrics = c("alpha", "kappa", "ac1", "pabak", "percent"),
  units = c("intersection", "union"),
  by_code = TRUE,
  align = c("grid", "presence", "coverage"),
  codebook_authors = NULL,
  codebook = NULL
)

Arguments

Value

A list with overall, by_code (NULL when by_code is FALSE), disagree, and independent (NULL unless codebook_authors is supplied and at least two other coders remain). Presence alignment adds codes_left_out, the number of codes never applied on a shared unit; these are excluded from by_code and the overall means, since a code absent from every shared unit would otherwise score perfect agreement. For grid alignment these are the metric, per-code, and disagreement tables; for coverage they hold per-coder-pair span coverage and the uncovered spans. In the metric tables n counts the units each statistic was computed on: units carrying a code from every coder for percent, pabak, and kappa; units carrying a code from at least two for ac1 and alpha. n is NA wherever the estimate is.

See Also

apply_codes() to produce assignments; merge_codes() to combine coders; code_retest() for AI stability.

Description

Returns the settings recorded by apply_codes() as a one-row tibble: provider, model, temperature, prompt and codebook hashes, unit, package version, and run time. Pass a model id with a dated snapshot (for example "gpt-4o-2024-08-06") to apply_codes() so the run can be repeated.

Usage

code_provenance(x)

Arguments

Value

A one-row tibble.

Description

Codes the same sample more than once at identical settings and reports how often the runs agree, against a shuffled-label baseline. Low temperature does not guarantee stable output, so stability is measured, not assumed.

Usage

code_retest(texts, codebook, n_runs = 2, sample_n = 50, seed = 123, ...)

Arguments

Value

A list with summary (tibble of metric, estimate, n_units, n_runs) and runs (all assignments, with a run column), or invisible(NULL) when fewer than two runs complete.

See Also

apply_codes(); code_agreement() for human inter-coder reliability, which this does not replace.

Description

Counts distinct codes after each unit is coded, in coding order. A curve with no new code in the last window units is evidence the codebook stopped learning; a rising curve means new codes still appear.

Usage

code_saturation(assignments, window = 10)

Arguments

Value

A list with curve (step, unit_id, new_codes, distinct_codes) and summary (n_units, n_codes, last_new_step, units_since_new, window, saturated).

See Also

round_summary() for the uncoded share across rounds.

Description

Groups codes under the themes named in the codebook and counts the coded units and documents each theme covers. Codes without a theme fall under "(no theme)". Themes are the researcher's grouping; nothing is inferred.

Usage

code_themes(assignments, codebook)

Arguments

Value

A tibble with theme, codes, n_codes, n_units, and n_docs.

See Also

code_saturation() for the code accumulation curve.

Description

Unified dispatcher for image description using vision LLMs. Routes to the appropriate provider (OpenAI or Gemini).

Usage

describe_image(
  image_base64,
  provider = "gemini",
  model = NULL,
  api_key = NULL,
  prompt =
    "Describe this image: charts, diagrams, tables, and text. Extract visible text.",
  timeout = 120,
  mime_type = "image/png"
)

Arguments

Value

Character string description, or NULL on failure

Description

Scores texts against snowball stopword lists and ranks candidate languages by the share of tokens matching each list. Uses only the stopwords package, so no language-detection dependency is added. Accuracy is good for European languages given a few hundred tokens, and poor for very short texts or languages absent from snowball; treat the result as a suggestion.

Usage

detect_language(texts, languages = NULL, sample_n = 200, seed = 123)

Arguments

Value

A tibble of language and score (share of tokens matching that language's stopwords), ranked best first, or NULL when no tokens are found. Ties and low scores mean the corpus language is unclear.

See Also

prep_texts() for the stopwords_language argument this informs.

Description

Identifies the dominant language of a text sample by asking an LLM (OpenAI or Gemini) to pick one name from a candidate list. More accurate than detect_language() on short texts, mixed-language corpora, or languages with sparse snowball stopword coverage, at the cost of an API call.

Usage

detect_language_llm(
  texts,
  languages,
  provider = c("auto", "openai", "gemini", "ollama"),
  model = NULL,
  api_key = NULL,
  sample_n = 200,
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A one-row tibble with language (code) and provider, or NULL when no provider/key is available or the response doesn't match a candidate.

See Also

detect_language() for the local, no-API alternative; call_llm_api() for the direct provider call.

Examples

if (interactive()) {
detect_language_llm(
  c("Bonjour, comment allez-vous?", "Je suis ravi de vous rencontrer."),
  languages = c(English = "en", French = "fr", Spanish = "es"),
  provider = "openai",
  api_key = Sys.getenv("OPENAI_API_KEY")
)
}

Description

This function detects multi-word expressions (collocations) of specified sizes that appear at least a specified number of times in the provided tokens.

Usage

detect_multi_words(tokens, size = 2:5, min_count = 2)

Arguments

Value

A character vector of detected collocations.

Examples

mydata <- TextAnalysisR::SpecialEduTech[seq_len(50), ]

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  collocations <- TextAnalysisR::detect_multi_words(tokens, size = 2:3, min_count = 2)
  print(collocations)

Description

Analyzes PDF to determine if it contains readable text.

Usage

detect_pdf_content_type(file_path)

Arguments

Details

Attempts text extraction using pdftools. Returns "text" if successful, or "unknown" if extraction fails or PDF is empty.

For table extraction from PDFs, use extract_tables_from_pdf_py.

Value

Character string: "text" or "unknown"

Examples

if (interactive()) {
  content_type <- detect_pdf_content_type(file.choose())
  print(content_type)
}

Description

Words that describe special-education and statistical topics rather than an evaluation (such as "disability", "intervention", "significant"). Pass the result to neutral_terms in sentiment_lexicon_analysis() or sentiment_valence_analysis() to leave them unscored.

Usage

domain_neutral_terms()

Value

Character vector of words.

Examples

head(domain_neutral_terms())

Description

Draws the confirmation sample only from units no one opened during refinement, so confirmation tests the codebook on units it was not built from.

Usage

draw_holdout(unit_ids, units_read = NULL, n = 50, strata = NULL, seed = 2026)

Arguments

Value

A character vector of unit identifiers. When fewer than n units remain, all of them are returned with a warning.

See Also

log_units_read(), code_agreement().

Description

Removes near-duplicate articles, which database exports carry whenever syndicated copy runs in several outlets. Comparison is across the whole data frame, so this runs after all files are combined; one file at a time can never reveal a duplicate.

Usage

drop_duplicate_articles(
  df,
  text_col = "text",
  date_col = "date",
  threshold = 0.99
)

Arguments

Value

df with later copies of each duplicated article removed. Returns df unchanged when it holds fewer than two rows or the comparison fails.

See Also

import_news_export(), which calls this when dedupe = TRUE

Description

Estimates how a document-level covariate shifts topic prevalence, returning a tidy data frame of per-topic estimates with intervals.

Usage

estimate_topic_effects(
  estimates,
  variable,
  type = c("pointestimate", "continuous"),
  method = c("stm", "beta"),
  interval = NULL,
  model = NULL,
  documents = NULL,
  npoints = 100,
  nsims = 100,
  ci = 0.95
)

Arguments

Value

A data frame with columns topic, value, proportion, lower, and upper.

Description

Export document clustering analysis results to CSV

Usage

export_document_clustering(
  coordinates,
  clusters = NULL,
  labels = NULL,
  doc_ids = NULL,
  file_path
)

Arguments

Value

Invisible data frame of the exported data

Description

Given a full similarity matrix and category information, extracts pairwise similarities between a reference category and other categories into a long-format data frame suitable for visualization and analysis.

Usage

extract_cross_category_similarities(
  similarity_matrix,
  docs_data,
  reference_category,
  compare_categories = NULL,
  category_var = "category",
  id_var = "display_name",
  name_var = NULL
)

Arguments

Value

A data frame with columns:

Examples

articles <- TextAnalysisR::SpecialEduTech[1:6, ]
articles$display_name <- paste0("d", seq_len(nrow(articles)))
term_matrix <- as.matrix(quanteda::dfm(quanteda::tokens(articles$abstract)))
normalized_matrix <- term_matrix / sqrt(rowSums(term_matrix ^ 2))
similarity_matrix <- normalized_matrix %*% t(normalized_matrix)
dimnames(similarity_matrix) <- list(articles$display_name, articles$display_name)
cross_similarities <- extract_cross_category_similarities(
  similarity_matrix  = similarity_matrix,
  docs_data          = articles,
  reference_category = "thesis",
  compare_categories = "journal_article",
  category_var       = "reference_type",
  id_var             = "display_name",
  name_var           = "title"
)

Description

Extracts distinctive keywords by comparing a target document group with the rest, reporting significance (G-squared by default, Benjamini-Hochberg adjusted p-value) alongside an effect size (Log Ratio).

Usage

extract_keywords_keyness(
  dfm,
  target,
  top_n = 20,
  measure = "lr",
  min_count = 0,
  rank_by = c("score", "log_ratio"),
  significant_only = FALSE
)

Arguments

Value

Data frame with columns: Keyword, Keyness_Score, Target_Count, Reference_Count, Target_per_10k, Reference_per_10k, Log_Ratio (log2 ratio of normalized frequencies; zero counts set to 0.5), and P_Adjusted (Benjamini-Hochberg). A non-empty result carries the keyness measure in attr(, "measure"), which plot_keyness_keywords() reads for its labels.

References

Hardie, A. (2014). Log Ratio: an informal introduction. ESRC Centre for Corpus Approaches to Social Science (CASS), Lancaster University.

Gabrielatos, C. (2018). Keyness analysis: nature, metrics and techniques. In C. Taylor & A. Marchi (Eds.), Corpus Approaches to Discourse (pp. 225-258). Routledge.

Examples

abstracts <- TextAnalysisR::SpecialEduTech$abstract[1:10]
dfm_object <- quanteda::dfm(quanteda::tokens(quanteda::corpus(abstracts)))
keywords <- extract_keywords_keyness(dfm_object, target = 1)
print(keywords)

Description

Extracts top keywords from a document-feature matrix using TF-IDF weighting. Uses quanteda's default scheme (raw term counts, base-10 idf), which differs from tidytext's proportion-based tf; rankings favor terms in longer documents.

Usage

extract_keywords_tfidf(dfm, top_n = 20, normalize = FALSE)

Arguments

Value

Data frame with columns: Keyword, TF_IDF_Score, Frequency

Examples

library(quanteda)
  corp <- corpus(c("text analysis", "data mining", "text mining"))
  dfm_obj <- dfm(tokens(corp))
  keywords <- extract_keywords_tfidf(dfm_obj, top_n = 5)
  print(keywords)

Description

Uses spaCy to extract morphological features from text. Returns data with Number, Tense, VerbForm, Person, Case, Mood, Aspect, etc.

Usage

extract_morphology(
  tokens,
  features = c("Number", "Tense", "VerbForm", "Person", "Case", "Mood", "Aspect"),
  include_pos = TRUE,
  include_lemma = TRUE,
  model = "en_core_web_sm"
)

Arguments

Details

Morphological features follow Universal Dependencies annotation. Common features include:

• Number: Sing (singular), Plur (plural)

• Tense: Past, Pres (present), Fut (future)

• VerbForm: Fin (finite), Inf (infinitive), Part (participle), Ger (gerund)

• Person: 1, 2, 3 (first, second, third person)

• Case: Nom (nominative), Acc (accusative), Gen (genitive), Dat (dative)

• Mood: Ind (indicative), Imp (imperative), Sub (subjunctive)

• Aspect: Perf (perfective), Imp (imperfective), Prog (progressive)

Value

A data frame with token-level morphological annotations including morph_* columns for each requested feature.

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  morphology_data <- extract_morphology(tokens)
  print(morphology_data)
}

Description

Uses spaCy to extract named entities (NER) from tokenized text. Returns a data frame with token-level entity annotations.

Usage

extract_named_entities(
  tokens,
  include_pos = TRUE,
  include_lemma = TRUE,
  model = "en_core_web_sm"
)

Arguments

Details

This function requires the Python with spaCy installed. If spaCy is not initialized, this function will attempt to initialize it with the specified model.

Value

A data frame with columns:

• doc_id: Document identifier

• token: Original token

• entity: Named entity type (e.g., PERSON, ORG, GPE)

• pos: Universal POS tag (if include_pos = TRUE)

• lemma: Lemmatized form (if include_lemma = TRUE)

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  entity_data <- extract_named_entities(tokens)
  print(entity_data)
}

Description

Extract noun chunks (base noun phrases) from texts. Useful for keyphrase extraction.

Usage

extract_noun_chunks(x, model = "en_core_web_sm")

Arguments

Value

A data frame with noun chunk information.

Description

Uses spaCy to extract part-of-speech (POS) tags from tokenized text. Returns a data frame with token-level POS annotations.

Usage

extract_pos_tags(
  tokens,
  include_lemma = TRUE,
  include_entity = FALSE,
  include_dependency = FALSE,
  model = "en_core_web_sm"
)

Arguments

Details

This function requires the Python with spaCy installed. If spaCy is not initialized, this function will attempt to initialize it with the specified model.

Value

A data frame with columns:

• doc_id: Document identifier

• sentence_id: Sentence number within document

• token_id: Token position within sentence

• token: Original token

• pos: Universal POS tag (e.g., NOUN, VERB, ADJ)

• tag: Detailed POS tag (e.g., NN, VBD, JJ)

• lemma: Lemmatized form (if include_lemma = TRUE)

• entity: Named entity type (if include_entity = TRUE)

• head_token_id: Head token in dependency tree (if include_dependency = TRUE)

• dep_rel: Dependency relation type, e.g., nsubj, dobj (if include_dependency = TRUE)

Examples

if (interactive()) {
  tokens <- quanteda::tokens(TextAnalysisR::SpecialEduTech$abstract[1])
  pos_data <- extract_pos_tags(tokens)
  print(pos_data)
}

Description

Extract subject-verb-object (SVO) triples from texts using dependency parsing.

Usage

extract_subjects_objects(x, model = "en_core_web_sm")

Arguments

Value

A data frame with SVO information.

Description

Unified helper that produces the long-format data.frame(topic, term, beta) expected by generate_topic_labels() from an STM model or an embedding result. Dispatches on the object's structure:

• STM model (has ⁠$beta$logbeta⁠ and ⁠$vocab⁠) -> top terms via stm::labelTopics() FREX

• Embedding result (has ⁠$topic_keywords⁠) -> c-TF-IDF keywords with rank-derived pseudo-beta

Usage

extract_topic_terms_df(model, n = 7)

Arguments

Value

data.frame(topic, term, beta) in long format.

Description

Searches for the optimal number of topics (K) using stm::searchK. Produces diagnostic plots to help select the best K value. The default spectral initialization is deterministic; with LDA or random initialization, fit several seeds per K before selecting a model.

Usage

find_optimal_k(
  dfm_object,
  topic_range,
  max.em.its = 75,
  emtol = 1e-04,
  cores = 1,
  categorical_var = NULL,
  continuous_var = NULL,
  init.type = "Spectral",
  height = 600,
  width = 800,
  seed = 123,
  verbose = TRUE,
  ...
)

Arguments

Value

A list with the search results, the model call, and the settings used.

See Also

plot_quality_metrics() to visualize topic-count diagnostics; stm::stm() to fit the chosen model; fit_embedding_model() for an embedding-based alternative to STM

Description

Find words most similar to a given word using word vectors. Requires a spaCy model with word vectors (en_core_web_md or en_core_web_lg).

Usage

find_similar_words(word, top_n = 10L, model = "en_core_web_md")

Arguments

Value

A data frame with similar words and similarity scores.

Description

This function finds the most similar topics to a given query using semantic similarity analysis. It works with both semantic topic models and traditional STM models by creating topic representations using transformer embeddings and calculating cosine similarity scores.

Usage

find_topic_matches(
  topic_model,
  query,
  top_n = 10,
  method = "cosine",
  embedding_model = "all-MiniLM-L6-v2",
  include_terms = TRUE
)

Arguments

Value

A list containing similar topics and their similarity scores.

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )
  texts <- united_tbl$united_texts

  topic_model <- TextAnalysisR::fit_embedding_model(
    texts = texts,
    method = "umap_hdbscan",
    n_topics = 8
  )

  similar_topics <- TextAnalysisR::find_topic_matches(
    topic_model = topic_model,
    query = "mathematical learning",
    top_n = 5
  )

  print(similar_topics)
}

Description

This function performs embedding-based topic modeling using transformer embeddings and specialized clustering techniques. Supports two backends:

• Python backend (default): Uses BERTopic library which combines transformer embeddings with UMAP dimensionality reduction and HDBSCAN clustering for optimal topic discovery.

• R backend: Uses R-native packages (umap, dbscan, Rtsne) for users without Python/BERTopic installed. Provides similar functionality with c-TF-IDF keyword extraction.

Usage

fit_embedding_model(
  texts,
  method = "umap_hdbscan",
  n_topics = NULL,
  embedding_model = "all-MiniLM-L6-v2",
  backend = "auto",
  clustering_method = "kmeans",
  similarity_threshold = 0.7,
  min_topic_size = 10,
  min_cluster_size = NULL,
  cluster_selection_method = "eom",
  umap_neighbors = 15,
  umap_min_dist = 0,
  umap_n_components = 5,
  umap_metric = "cosine",
  tsne_perplexity = 30,
  pca_dims = 50,
  dbscan_eps = 0.5,
  dbscan_minpts = 5,
  representation_method = "c-tfidf",
  diversity = 0.5,
  reduce_outliers = FALSE,
  outlier_strategy = "probabilities",
  outlier_threshold = 0,
  seed = 123,
  verbose = TRUE,
  precomputed_embeddings = NULL,
  stop_words = NULL
)

Arguments

Value

A list containing topic assignments, topic keywords, and quality metrics.

See Also

get_best_embeddings() to supply precomputed embeddings; generate_topic_labels() for AI-suggested topic names; find_optimal_k() for an STM-based alternative

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )
  texts <- united_tbl$united_texts

  # Embedding-based topic modeling (powered by BERTopic)
  result <- TextAnalysisR::fit_embedding_model(
    texts = texts,
    method = "umap_hdbscan",
    n_topics = 8,
    min_topic_size = 3
  )

  print(result$topic_assignments)
  print(result$topic_keywords)
}

Description

Performs semantic analysis including similarity, dimensionality reduction, and clustering in a single call.

Usage

fit_semantic_model(
  texts,
  analysis_types = c("similarity", "dimensionality_reduction", "clustering"),
  document_feature_type = "embeddings",
  similarity_method = "cosine",
  use_embeddings = TRUE,
  embedding_model = "all-MiniLM-L6-v2",
  dimred_method = "UMAP",
  clustering_method = "umap_dbscan",
  n_components = 2,
  n_clusters = 5,
  seed = 123,
  verbose = TRUE
)

Arguments

Value

A list containing results from requested analyses.

Examples

if (interactive()) {
  data(SpecialEduTech)
  texts <- SpecialEduTech$abstract[1:10]

  results <- fit_semantic_model(
    texts = texts,
    analysis_types = c("similarity", "clustering")
  )

  print(results)
}

Description

Analyzes how topics evolve over time by fitting topic models to different time periods and tracking semantic changes.

Usage

fit_temporal_model(
  texts,
  dates,
  time_windows = "yearly",
  embeddings = NULL,
  verbose = TRUE
)

Arguments

Value

A list containing temporal analysis results with topic evolution patterns.

Description

Suggests descriptive labels for clusters using AI. Labels are suggestions for human review - users should edit and approve before using. Supports OpenAI or Gemini for AI generation.

Usage

generate_cluster_labels(
  cluster_keywords,
  provider = "auto",
  model = NULL,
  temperature = 0.3,
  max_tokens = 50,
  api_key = NULL,
  verbose = TRUE
)

Arguments

Value

A list of generated labels.

Examples

if (interactive()) {
  cluster_keywords <- list(
    "1" = c("calculator", "arithmetic", "elementary", "remedial"),
    "2" = c("computer", "instruction", "multiplication", "drill")
  )
  labels_openai <- generate_cluster_labels(cluster_keywords, provider = "openai")
  labels_gemini <- generate_cluster_labels(cluster_keywords, provider = "gemini")
}

Description

Generate descriptive labels for document clusters

Usage

generate_cluster_labels_auto(
  feature_matrix,
  clusters,
  method = "tfidf",
  n_terms = 3
)

Arguments

Value

Named list of cluster labels

Description

Reads one unit of text at a time with the codebook so far in view, and either reuses a code already in it or opens a new one. The codebook is an output here, not an input as in apply_codes(). Supplying codebook seeds the reading, which also covers starting from a priori codes or continuing a stopped run. Categories are proposed for human review, not settled.

Usage

generate_codes(
  texts,
  codebook = NULL,
  unit = c("sentence", "paragraph", "document"),
  order = NULL,
  context_k = Inf,
  stop_after = Inf,
  provider = c("auto", "openai", "gemini", "ollama"),
  model = NULL,
  temperature = 0,
  api_key = NULL,
  max_tokens = 200,
  delay = 1,
  verbose = TRUE
)

Arguments

Value

A list with codebook (tibble of code, definition, first_unit) and trace (one row per unit read: position, doc_id, unit_id, assigned, opened_new, n_categories, status), one row per unit read in reading order. The trace shows where categories stopped accumulating. Returns invisible(NULL) when no provider key is available.

See Also

apply_codes() to code a corpus once the codebook is settled; uncoded_units() for what a codebook does not reach.

Description

Generates L2-normalized embeddings for texts using sentence transformers.

Usage

generate_embeddings(texts, model = "all-MiniLM-L6-v2", verbose = TRUE)

Arguments

Value

A matrix of L2-normalized embeddings.

Description

Generate keywords for topics using c-TF-IDF (class-based TF-IDF), similar to BERTopic. This method treats all documents in a topic as a single document and calculates TF-IDF scores relative to other topics.

Usage

generate_semantic_topic_keywords(
  texts,
  topic_assignments,
  n_keywords = 10,
  method = "c-tfidf",
  diversity = 0.5
)

Arguments

Value

A list of keywords for each topic.

Description

Uses Large Language Models (LLMs) to generate various types of content based on topic model terms. Supports multiple content types with optimized default prompts, or fully custom prompts.

Usage

generate_topic_content(
  topic_terms_df,
  content_type = c("survey_item", "research_question", "theme_description",
    "policy_recommendation", "interview_question", "custom"),
  topic_var = "topic",
  term_var = "term",
  weight_var = "beta",
  provider = c("openai", "gemini", "ollama"),
  model = NULL,
  temperature = 0,
  system_prompt = NULL,
  user_prompt_template = NULL,
  max_tokens = 150,
  api_key = NULL,
  output_var = NULL,
  verbose = TRUE
)

Arguments

Details

The function generates one piece of content per unique topic. Each content type has optimized default prompts, but these can be overridden with custom prompts.

Requires an API key set via the api_key parameter or the relevant environment variable (OPENAI_API_KEY or GEMINI_API_KEY).

Value

A data frame with generated content joined to original topic terms.

See Also

generate_topic_labels() for the step that creates topic labels; get_content_type_prompt() and get_content_type_user_template() to inspect or override default prompts

Examples

if (interactive()) {
# Generate survey items
survey_items <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "survey_item",
  provider = "openai",
  model = "gpt-4.1-mini"
)

# Generate research questions
research_qs <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "research_question",
  provider = "gemini",
  model = "gemini-3.5-flash-lite"
)

# Generate with custom prompt
custom_content <- generate_topic_content(
  topic_terms_df = top_terms,
  content_type = "custom",
  system_prompt = "You are an expert in educational policy...",
  user_prompt_template = "Based on {terms}, generate a learning objective:"
)
}

Description

This function generates descriptive labels for each topic based on their top terms using AI providers (OpenAI or Gemini).

Usage

generate_topic_labels(
  top_topic_terms,
  provider = "auto",
  model = NULL,
  system = NULL,
  user = NULL,
  temperature = 0.5,
  api_key = NULL,
  openai_api_key = NULL,
  verbose = TRUE
)

Arguments

Value

A data frame containing the top terms for each topic along with their generated labels. The "llm" attribute records the provider, model, and temperature.

See Also

get_topic_terms() to extract top terms first; generate_topic_content() for survey items / RQs / themes grounded in the same terms; call_llm_api() for the direct AI provider call

Examples

if (interactive()) {
top_topic_terms <- get_topic_terms(stm_model, top_term_n = 10)

# Auto-detect provider (tries OpenAI -> Gemini)
labels <- generate_topic_labels(top_topic_terms)

# Use specific provider
labels_openai <- generate_topic_labels(top_topic_terms, provider = "openai")
labels_gemini <- generate_topic_labels(top_topic_terms, provider = "gemini")
}

Description

Auto-detects and uses the best available embedding provider with the following priority:

1. sentence-transformers (local Python) - if Python environment is set up

2. OpenAI API - if OPENAI_API_KEY is set

3. Gemini API - if GEMINI_API_KEY is set

Usage

get_best_embeddings(
  texts,
  provider = "auto",
  model = NULL,
  api_key = NULL,
  verbose = TRUE
)

Arguments

Value

Matrix with embeddings (rows = texts, columns = dimensions)

Examples

if (interactive()) {
data(SpecialEduTech)
texts <- SpecialEduTech$abstract[1:5]

# Auto-detect best available provider
embeddings <- get_best_embeddings(texts)

# Force specific provider
embeddings <- get_best_embeddings(texts, provider = "openai")

dim(embeddings)
}

Description

Returns the default system prompt for a given content type.

Usage

get_content_type_prompt(content_type)

Arguments

Value

Character string with the system prompt.

Description

Returns the default user prompt template for a given content type.

Usage

get_content_type_user_template(content_type)

Arguments

Value

Character string with the user prompt template containing {terms} placeholder.

Description

Returns memos filtered by target and round. Omitted filters match everything, so calling with no filter returns the full table in the order written.

Usage

get_memos(memos, target_type = NULL, target_id = NULL, round = NULL)

Arguments

Value

A tibble of matching memos, oldest first.

See Also

add_memo() to write them.

Description

Segment texts into sentences using spaCy's sentence boundary detection.

Usage

get_sentences(x, model = "en_core_web_sm")

Arguments

Value

A data frame with sentence information.

Description

Get information about the currently loaded spaCy model.

Usage

get_spacy_model_info()

Value

A list with model information including name, language, pipeline components, and vector availability.

Description

Extracts topic prevalence values (gamma/theta) from a fitted STM model, returning mean prevalence for each topic as a data frame.

Usage

get_topic_prevalence(stm_model, category = NULL, include_theta = FALSE)

Arguments

Value

A data frame with columns:

Examples

if (interactive() && requireNamespace("stm", quietly = TRUE)) {
  # Requires fitting an STM model first; uses 'stm::gadarian' for demo
  data("gadarian", package = "stm")
  proc <- stm::textProcessor(gadarian$open.ended.response, metadata = gadarian)
  prep <- stm::prepDocuments(proc$documents, proc$vocab, proc$meta)
  topic_model <- stm::stm(prep$documents, prep$vocab, K = 3,
                           data = prep$meta, max.em.its = 5,
                           verbose = FALSE)
  prevalence <- get_topic_prevalence(topic_model)
  prevalence_label <- get_topic_prevalence(topic_model, category = "demo")
}

Description

This function selects the top terms for each topic based on their word probability distribution (beta).

Usage

get_topic_terms(stm_model, top_term_n = 10, verbose = TRUE, ...)

Arguments

Value

A data frame containing the top terms for each topic.

Examples

if (interactive() && requireNamespace("stm", quietly = TRUE)) {
  mydata <- TextAnalysisR::SpecialEduTech

  united_tbl <- TextAnalysisR::unite_cols(
    mydata,
    listed_vars = c("title", "keyword", "abstract")
  )

  tokens <- TextAnalysisR::prep_texts(united_tbl, text_field = "united_texts")

  dfm_object <- quanteda::dfm(tokens)

  out <- quanteda::convert(dfm_object, to = "stm")

  stm_15 <- stm::stm(
    data = out$meta,
    documents = out$documents,
    vocab = out$vocab,
    max.em.its = 75,
    init.type = "Spectral",
    K = 15,
    prevalence = ~ reference_type + s(year),
    verbose = TRUE
  )

  top_topic_terms <- TextAnalysisR::get_topic_terms(
    stm_model = stm_15,
    top_term_n = 10,
    verbose = TRUE
  )
  print(top_topic_terms)
}

Description

Concatenates top terms for each topic into text strings suitable for embedding generation. Useful for creating topic representations for semantic similarity analysis.

Usage

get_topic_texts(
  top_terms_df,
  topic_var = "topic",
  term_var = "term",
  weight_var = NULL,
  sep = " ",
  top_n = NULL
)

Arguments

Value

A character vector of topic text strings, one per topic, ordered by topic number.

Examples

# Topic-term frame as produced by get_topic_terms()
top_terms <- data.frame(
  topic = c(1, 1, 1, 2, 2, 2),
  term  = c("calculator", "arithmetic", "elementary",
            "computer", "instruction", "multiplication"),
  prob  = c(0.10, 0.08, 0.07, 0.12, 0.09, 0.06)
)
get_topic_texts(top_terms)
get_topic_texts(top_terms, weight_var = "prob", top_n = 2)

Description

Calculate semantic similarity between two words using word vectors. Requires a spaCy model with word vectors (en_core_web_md or en_core_web_lg).

Usage

get_word_similarity(word1, word2, model = "en_core_web_md")

Arguments

Value

A list with similarity score and metadata.

Description

Summarizes each group from an unsupervised grouping for a researcher to rule on: how tightly its units sit around their center (coherence), which other group sits closest and how close (separation), units drawn at random inside the group, its furthest member, and the closest group's units that fit this group best. Random draws rather than the most central units are shown, because the center is a biased sample of a group.

Usage

group_evidence(
  embeddings,
  groups,
  unit_ids = NULL,
  n_random = 3,
  n_neighbor = 2,
  hide = NULL,
  seed = 2026
)

Arguments

Value

A tibble with one row per group: group, n, coherence, nearest, separation, and list columns random_units, boundary_unit, and neighbor_units holding unit identifiers. Separation above coherence raises the question of a merge; the units answer it.

See Also

apply_ruling() to act on the verdicts.

Description

Identifies trending topics in temporal results

Usage

identify_topic_trends(temporal_results, ...)

Arguments

Value

List containing identified trends

Description

This function processes different types of files and text input based on the dataset choice.

Usage

import_files(
  dataset_choice,
  file_info = NULL,
  text_input = NULL,
  pdf_unit = c("line", "page", "document")
)

Arguments

Value

A data frame containing the processed data. PDF rows carry page and page_end, the first and last PDF page of the row's text, so a coded quote stays traceable to its source page. Other formats have no page columns; combined with PDFs, their rows get NA. When several files are imported, each row carries a file column with the source file name (file_info$name when present, otherwise the base name of filepath), unless the file already has a file column. Two-column PDF pages are split at the column gutter and read left column first.

Examples

if (interactive()) {
  mydata <- TextAnalysisR::SpecialEduTech
  mydata <- TextAnalysisR::import_files(dataset_choice = "Upload an Example Dataset")
  head(mydata)

  xlsx_path <- system.file("extdata", "SpecialEduTech.xlsx",
                           package = "TextAnalysisR")
  file_info <- data.frame(filepath = xlsx_path)
  mydata <- TextAnalysisR::import_files(dataset_choice = "Upload Your File",
                                          file_info = file_info)
  head(mydata)


  text_input <- paste("Virtual manipulatives for algebra instruction",
                      "manipulatives mathematics learning disability",
                      "This study examined virtual manipulatives effects on",
                      "students with learning disabilities")
  mydata <- TextAnalysisR::import_files(dataset_choice = "Copy and Paste Text",
                                          text_input = text_input)
  head(mydata)
}

Description

Converts image files into text documents by sending each image to a vision model and keeping the returned description and transcribed text as the document body. One row per image.

Usage

import_images(
  file_paths,
  provider = "gemini",
  model = NULL,
  api_key = NULL,
  prompt = NULL,
  labels = NULL
)

Arguments

Value

A tibble with text and category columns. Images the model could not describe are dropped.

See Also

describe_image() for the single-image call; import_files() for text formats

Description

Parses a LexisNexis-style export with LexisNexisTools, returning one row per article with the body text separated from the headline, outlet, date, section, and author. Headline and outlet stay as their own fields rather than entering the body, so they can group or filter documents without inflating word counts.

Usage

import_news_export(file_paths, dedupe = FALSE, threshold = 0.99)

Arguments

Value

A data frame with text, category, headline, newspaper, date, section, and author, or NULL when nothing parses as an export.

See Also

import_files() for ordinary documents; remove_metadata_lines() for exports this cannot parse

Description

Converts tokens to their lemmatized forms using spaCy, with batch processing to handle large document collections without timeout issues.

Usage

lemmatize_tokens(
  tokens,
  batch_size = 50,
  model = "en_core_web_sm",
  verbose = TRUE
)

Arguments

Details

Uses spaCy for linguistic lemmatization producing proper dictionary forms (e.g., "studies" -> "study", "better" -> "good"). Batch processing prevents timeout errors with large document collections. If any batch fails, the function stops rather than returning a shorter corpus.

Value

A quanteda tokens object containing lemmatized tokens, in the original document order and with the original document variables.

Examples

if (interactive()) {
tokens <- quanteda::tokens(c("The studies showed better results"))
lemmatized <- lemmatize_tokens(tokens, batch_size = 50)
}

Description

Functions for lexical analysis including:

• Linguistic Annotation (POS tagging, NER)

• Frequency Analysis (word frequency, n-grams, MWEs)

• Keywords (TF-IDF, keyness)

• Lexical Diversity (TTR, MTLD, MATTR)

• Readability (Flesch, Gunning Fog, etc.)

Description

Calculates multiple lexical diversity metrics for a document-feature matrix (DFM) or tokens object. Supports all quanteda.textstats measures plus MTLD (Measure of Textual Lexical Diversity), which is the most recommended measure according to McCarthy & Jarvis (2010) for being independent of text length.

Usage

lexical_diversity_analysis(
  x,
  measures = "all",
  texts = NULL,
  cache_key = NULL,
  window = 50
)

Arguments

Details

MTLD (Measure of Textual Lexical Diversity) is calculated using the algorithm from McCarthy & Jarvis (2010). It counts the number of "factors" needed to reduce TTR below 0.72, then divides the number of tokens by the number of factors. This provides a length-independent measure of lexical diversity. A factor closes only once it spans at least 10 tokens, so early TTR dips in the first few tokens do not count; this floor is a package choice, not part of the original McCarthy and Jarvis (2010) algorithm. Documents under 10 tokens, or where no factor completes in either direction, return NA.

Important notes:

• For MTLD accuracy, pass a tokens object (not DFM) as input

• If using DFM, provide the 'texts' parameter for MTLD calculation

• MATTR and MSTTR use one fixed window; documents shorter than it return NA

• TTR, C, R, CTTR, Maas, K, and D depend on text length (TTR falls mechanically as documents lengthen); compare them only across documents of similar n_tokens

• MTLD and MATTR are most reliable at 100+ tokens per document

• Results are cached when cache_key is provided for repeated analysis

Value

A list containing:

• lexical_diversity: Data frame with per-document lexical diversity scores and n_tokens, the number of tokens per document

• summary_stats: List of summary statistics (mean, median, sd) for each measure

References

McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381-392.

Covington, M. A., & McFall, J. D. (2010). Cutting the Gordian knot: The moving-average type-token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94-100.

See Also

calculate_text_readability() for grade-level / Flesch metrics on the same input; calculate_lexical_dispersion() for term spread across documents; plot_lexical_diversity_distribution() to visualize

Examples

data(SpecialEduTech)
texts <- SpecialEduTech$abstract[1:10]
corp <- quanteda::corpus(texts)
toks <- quanteda::tokens(corp)
# Preferred: pass tokens object for accurate MTLD
lex_div <- lexical_diversity_analysis(toks, texts = texts)
# With caching for repeated analysis
if (requireNamespace("digest", quietly = TRUE)) {
  cache_key <- digest::digest(texts)
  lex_div <- lexical_diversity_analysis(toks, texts = texts, cache_key = cache_key)
}
# Alternative: pass DFM with texts for MTLD accuracy
dfm_obj <- quanteda::dfm(toks)
lex_div <- lexical_diversity_analysis(dfm_obj, texts = texts)
print(lex_div)

Description

Plots word frequency for lexical analysis using plot_word_frequency.

Usage

lexical_frequency_analysis(...)

Arguments

Value

A plotly bar chart of word frequencies

Description

Reads a directory written by save_topic_model().

Usage

load_topic_model(path)

Arguments

Details

Load only folders from trusted sources: the BERTopic model is unpickled, which can run arbitrary code.

Value

The fit_embedding_model() result with bertopic_model restored.

See Also

save_topic_model(), transform_topic_model().

Description

Appends one row per pass through discovery, coding, and revision. The round number increments on its own, so repeated calls build the record in order.

Usage

log_round(
  rounds = NULL,
  n_categories,
  n_coded,
  n_uncoded,
  notes = NA_character_
)

Arguments

Value

The round table with one row appended.

See Also

round_summary() for the coverage trend across rounds; uncoded_units() for the count this records.

Description

Appends the units a researcher opened, so the report can state how much of the corpus refinement read and the holdout can exclude every unit seen.

Usage

log_units_read(units_read = NULL, unit_ids, round, step, coder)

Arguments