Skip to content

lang-chain

This journal groups my learning notes for the lang-chain framework.

References

Journal

Questions

  • In the conversation retrieval chain example, it was not clear how the MessagesPlaceholder mechanism used for chat_history is hooked up.
    • A MessagesPlaceholder is a list that will eventually contain LLM messages. The placeholder is filled later on via a variable defined by the creation of placeholder. It is like a PromptTemplate, but for a whole list of messages. Useful for injecting context and chat history.

Actions

2024-04-24

  • word2vec paper: recalled the bag-of-word and skip-gram models. The output of word2vec is a list of word vectors of dimension n, one for each word. The process that creates these word vectors are able to encode somehow syntactic (correspondent adjective of a noun); and semantic (the capital of a country is a delta vector in the embedding space) notions.
  • How attention works The attention mechanism is how the input embedding is modified in order to take into account the influence of the text into a word. Each attention head is composed of three matrices: Query, Key and Value; and each attention head captures somehow one aspect of the text that is relevant for the word (i.e. context, sentiment, logical reasoning)
    • The Query and Key matrices work together and their goal is to find the words in a given context that match the aspect that both matrices are supposed to encode (e.g. the query matrix may encode: is there any adjective around modifying me?; and the Key matrix could be encoding the notion of adjective.)
    • Words that are aligned in the sense of Query and Value will have a higher weight in the operation that follows. There is this Value matrix that is supposed to encode the necessary delta to be added to a word vector in order to approximate that word to some other word representing a concept X.
    • The GPT3 has 96 attention heads.
  • What does it mean to create an embedding for a text? I understand the framework in which we create embeddings for isolated words but what interpretation should I have for embeddings of texts? Is this embedding space representing the message meaning? How those embeddings are created? Are they created from word embeddings?

2024-04-19

  • Recall some of Linear Algebra concepts: Eigen decomposition and SVD.
  • SVD writes any linear transformation as the composition of: rotation, scaling and reflection.
  • SVD is the generalization of the eigen decomposition.
  • Watched a 3Blue1Brown video on Transfomers. It touched vector embeddings, the general structure of the multilayer neural network used to create the GPT model and softmax.
  • Took a glimpse in the FAISS documentation. Indeed, a vector store is all about computing \(argmin_{x} || x- x_i ||\) in a vector set \(x_i\) efficiently. This process is also called similarity search. FAISS is specialized on doing that for dense vectors.

2024-04-18

  • Start re-reading the wordvec paper to recall embeddings vocabulary.
  • Latent semantic analysis and Singular Value Decompsition.

2024-04-12

  • Langsmith is a platform to assist you in the development of LLM applications. It comes with several tools that will help you ship quickly and with confidence. It is free for a single user up to 5K traces a month.
  • OpenAI Embeddings: Module to create embeddings from OpenAI models. Be aware that they are charged.
  • OpenAI allows you to reduce the dimensionality of the created embeddings. The main advantage on this is to save storage space for example, in a vector database.
  • Applications of vector embeddings:
    • Document search
    • Recommendation
    • LLM prompt engineering: Retrieval prompts.
  • Implemented the Conversation Retrieval Chain example.

2024-04-11

  • Recall how word-embeddings are constructed. I have prepared a word-embedding study plan in the NLP journal. From TF-IDF, passing through word2vec, bert and transformers.
  • Read about vector stores and how we can use to retrieve relevant documents. A vector store is a database for vector embeddings.
  • FAISS (Facebook AI Similarity Search) is a vector store created by Facebook and it has a python library available.

2024-04-10

  • Created lang-chain-tutorial package.
  • Created a chain using a LLM backend (OpenAI), a prompt template and a output parser.
  • Created a prompt with a simple retrieval mechanism using the FAISS vectorstore.