Running it

If you only want the model serving, this is the setup I use daily: four Arc Pro B65 cards at tensor parallelism 4, the model’s full 256K context, and enough room configured for several requests in flight at once. A single stream decodes at 52.1 tok/s on an 8K prompt, and token-to-token latency stays flat as the context fills.

The weights are at ulkaa/Qwen3.8-27B-AWQ-INT4 and the pinned serving image is on Docker Hub as rahulunair/sglang-xpu, tag qwen3.8-27b-20260816.

docker run --rm --device=/dev/dri -v /dev/dri:/dev/dri \
  --group-add video --group-add "$(getent group render | cut -d: -f3)" \
  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
  --ipc=host --shm-size=64g --ulimit memlock=-1 \
  -p 30000:30000 -v /path/to/Qwen3.8-27B-AWQ-INT4:/model:ro \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  rahulunair/sglang-xpu:qwen3.8-27b-20260816 \
  python -m sglang.launch_server --model-path /model --device xpu \
    --tp-size 4 --attention-backend intel_xpu --page-size 64 \
    --context-length 262144 \
    --chunked-prefill-size 4096 --mem-fraction-static 0.85 \
    --cuda-graph-config '{"decode":{"backend":"full","bs":[1,2,4,8]},"prefill":{"backend":"disabled"}}'

Use --tp-size 2 on a two-card machine, where the same prompt decodes at 37.8 tok/s. Leaving --max-total-tokens unset lets the server size the KV pool from the memory it finds; pin it lower if you are sharing the cards with something else. The SYS_PTRACE and unconfined seccomp flags are there for one optional fast collective I describe later; drop both and the model still runs normally.

Do check current upstream SGLang and Intel images before reaching for a pinned image. A pinned release is useful for reproducing this article; it is not a reason to keep that patch set around forever.

The rest of this post is how I got there.

What are we actually running?

Qwen3.8-27B has 64 decoder layers. Forty-eight use Gated DeltaNet, a linear attention recurrence, and 16 use full attention. It is dense rather than mixture-of-experts, so almost every large text-model weight participates in every decode step. The checkpoint also carries a 27-layer vision tower and a multi-token-prediction head.

That shape splits the problem into two regimes:

• Decode at batch one has very little arithmetic reuse. It is mostly a question of how many bytes must cross memory for every new token, plus small kernels, layer boundaries, and collectives.
• Prefill turns the same projections into larger matrix operations and adds context-growing attention work. Compute throughput and tile efficiency matter much more here.

I kept two checkpoints working throughout.

1. The official Qwen3.8-27B-FP8 checkpoint, my reference while getting everything working.
2. My Qwen3.8-27B AWQ W4A16 checkpoint: 18.2 GiB, asymmetric group-128 quantization, BF16 vision tower and MTP tensors preserved.

“FP8 model” and “AWQ model” describe how a checkpoint stores its weights, not necessarily the instruction that runs every projection. The runtime may dequantize into BF16, keep an extra signed-INT8 copy for decode, or dispatch an actual four-bit primitive. Keeping storage precision, activation precision, and selected kernel separate in my head avoided a lot of confusion.

These are the final runs. Output was coherent, graph replay and the expected fast paths were active, and I used the second run of each shape. Earlier numbers helped me debug the model but are not useful baselines.

These are complete setups, not a one-change-at-a-time comparison: graph capture on, concurrency 1, second run, roughly 9K context at the end of decode.

The first two rows are the ones I keep coming back to. Two cards running my four-bit build land at 37.8 tok/s against 41.4 for the official checkpoint on four. The two rows use different checkpoint formats and different code paths, so an efficiency percentage between them would mean very little. What it does tell me is what this workload costs in hardware: two cards now give me the interactive feel I was getting from four.

The official FP8 checkpoint stays useful. It is the standard model, it works in the release image, and it helped me separate real model behaviour from bugs in my own quantization pipeline.

Before writing a kernel, do the boring math

The first question about any operator is whether it is limited by arithmetic or by memory traffic:

time >= max(useful_FLOPs / achievable_FLOPs_per_second,
            bytes_moved / achievable_bytes_per_second)

For batch-one dense decode, arithmetic intensity is close to one operation per weight byte. The memory term wins quickly, so the first useful model is just counting bytes:

bytes_per_token = sum(weights and metadata read by one decode step)
seconds_per_token = bytes_per_token / aggregate_achievable_bandwidth
tokens_per_second = 1 / seconds_per_token

Count from the safetensors headers, not from a number in config.json. An embedding table is resident but only one row gets looked up. The vision tower is resident but idle during a text-only turn. The MTP head is only read when speculation is active. On a multimodal model, counting every resident tensor can add several GiB the step never actually reads.

Quantization metadata counts too. A nominal four-bit weight with group scales and zero points is not exactly 0.5 bytes per parameter. In this format it is about 0.52 bytes at group 128 and about 0.59 at group 32, and that metadata is read alongside the payload on every token.

Prefill is a different calculation. A reasonable first approximation for the dense projections is 2 × parameters × prompt_tokens floating-point operations, with the attention term added separately. This is why one number called “model throughput” should never be used for both phases.

Intel lists both the B65 and the B70 at 32 GiB and 608 GB/s of memory bandwidth. The B65 has 20 Xe2 cores and 197 peak dense INT8 TOPS; the B70 has 32 cores and 367 TOPS. Same advertised bandwidth, roughly 1.86x the INT8 compute. Decode and prefill should therefore scale differently even before software enters the picture.

A roofline gives you a boundary. Mostly I use it as a warning system: it tells me when a proposed optimization cannot possibly repay its complexity, when a measurement looks suspiciously good, and when writing another kernel is unlikely to be worth the time.

Check upstream first, then add the missing bits

Pinned containers are good for reproducibility, and they also freeze the stack at one moment. The XPU stack kept moving while I worked, so my loop for every missing path became:

1. Check current SGLang, torch-xpu, oneDNN, Intel’s images, and the XPU kernel packages.
2. Confirm whether the failure is still present in the pinned release.
3. Separate a missing platform-specific branch from a missing implementation.
4. Add the smallest overlay that makes the model correct.
5. Delete the overlay when upstream covers the same case.

Step three is the one that saves weeks, and it came up immediately.

SGLang reads AWQ checkpoints through a component called compressed-tensors, which decides how packed four-bit weights get unpacked and which matrix kernel serves them. In the version I was using, two small things assumed an NVIDIA GPU. A helper that rearranges packed weights into the layout the kernel expects was imported only when CUDA was available, then called later regardless. Separately, the code that picks a quantization scheme asked CUDA for the device’s compute capability before it had chosen a scheme at all, which fails on a machine with no CUDA device.

Neither of those means Intel GPUs lack a four-bit kernel. The kernel was there. The checkpoint just could not reach it, because the road to it ran through NVIDIA-only code. Adding the missing branch was enough.

The MTP path had the same shape. I registered the XPU attention backend in the speculative draft maps, routed the token-tree convolution to a Triton implementation that accepts tree arguments, and relaxed two helpers that rejected non-CUDA tensors even though their implementation was already Triton. Small integration fixes, not a replacement serving stack.

One more lesson: “the model loaded” is only the beginning of correctness. My first text-only quantization build silently omitted the vision tower and the MTP head. The library never instantiated an MTP module, so save_pretrained could not save tensors it did not know existed. I now build with the multimodal class, copy the MTP tensors explicitly, and verify the tensor list before any performance run.

The existing library primitive was the fastest kernel I tested

Once the model was correct enough to benchmark, the obvious question was which four-bit matrix path should serve the dense projections.

The answer was not my hand-driven kernel. On the same community AWQ checkpoint, barrydeen/Qwen3.8-27B-AWQ-4bit, the same two B65 cards, and the same captured 128-input/512-output shape:

Figure 1. The existing oneDNN primitive, not a new custom kernel, produced the largest measured four-bit step on this shape.

That ATen primitive dispatches to oneDNN weight decompression. It needed no model-specific tuning file and, for this checkpoint layout, no second weight repack. Reading its source also explained a BF16/F16 difference I had measured: the specialized four-bit dequantization path in the oneDNN version I used only accepts F16 and F32 compute types, so BF16 falls through to a more generic tile-conversion branch.

I spent a day pushing the hand-driven path before accepting the stop condition: if the library primitive is already close to the in-situ roofline, keep it as the baseline and move up a level. oneDNN wins this round.

That was a large improvement, and it exposed the next problem. The community AWQ checkpoint left the three large Gated DeltaNet projections in BF16, and those represented nearly half the bytes read during a decode step. The kernel was no longer the only question. The checkpoint itself had become part of the latency path.

Four bits is not really four bits

My checkpoint inventory measured the three large projections in every DeltaNet layer (input QKV, input Z, and output) at roughly 10.36 GiB in BF16. That is about 47% of the community checkpoint’s decode traffic. I nearly left them alone. Then I looked at the official FP8 release and found scale tensors for those same projections, with only the small surrounding tensors excluded. Useful prior: the model authors quantize these in their own low-precision release.

My AWQ build quantizes 24.33 billion parameters with asymmetric group-128 W4A16. Per-group scales and zero points bring the stored cost of those weights to 4.16 bits per parameter. Another 3.45 billion parameters stay BF16: embeddings, output head, norms, small DeltaNet gates, vision tower, and MTP head. Across the full checkpoint that averages 5.63 bits per parameter, or 18.2 GiB.

I did not arrive at group 128 immediately. The first all-four-bit build used group 32. It had finer quantization groups and fewer total model bytes than the community checkpoint, and it ran slower. Two measurements explained the sign:

• Group 32 reads four times as many scale and zero-point groups.
• On the same down_proj tensor at M=1, oneDNN moved group-32 data at 383 GB/s and group-128 data at 505 GB/s. The matching BF16 GEMV reached 589 GB/s.

Group size turned out to be a serving-layout decision as much as a quality setting, and re-quantizing at 128 was worth more than another day of kernel changes.

Here the comparison is like for like: same cards, same TP2 setup, same 128-input/512-output shape, same server, two four-bit checkpoints of the same model. Quantizing the DeltaNet projections cut measured decode traffic from 21.82 GiB to 14.19 GiB, about 35%, and throughput moved from 35.4 to 38.5 tok/s.

About 9% faster than the community four-bit checkpoint. Thirty-five percent fewer bytes bought about nine percent more tokens per second, so the byte model found the opportunity, but those newly quantized projection shapes were clearly behaving unlike ordinary dense four-bit GEMMs.

I have not closed that gap. The merged DeltaNet projection at TP2 has a shape unlike the dense FFN projections and is my leading suspect, and I come back to it at the end. I would rather publish the result I cannot explain than tidy the story until the arithmetic looks neat.

Graph capture changes both performance and observability

Decode is a long chain of small operations. Graph replay removes much of the per-operation submission cost, so graph capture is part of the setup I actually use, not an optional benchmark trick. Earlier work on two other models in this series measured it as a first-order improvement. For Qwen3.8 I do not have a clean graph-on/graph-off comparison, so I attach no new percentage to it.

Capture also changes what instrumentation means. A Python-side counter that gets incremented while the graph is being captured can replay the device work forever without ever incrementing the Python value again. I once had a fast collective serving happily inside the graph while its own log insisted served=0.

The only trustworthy proof was controlled removal:

• build the same graph with the path enabled;
• build it again with exactly that path disabled;
• run the same shape after compilation;
• require the timing to move by more than the quiet-box noise floor.

One of those A/B tests was for a small collective operation. When the model is split across four cards, every layer has to sum partial results from all of them, an operation called an all-reduce. During decode that message is tiny: one token at hidden size 5,120 in BF16 is about 10 KiB. At that size almost none of the time goes into moving bytes. It goes into the fixed cost of setting up the exchange and getting the cards to agree they are ready.

So a leaner path for small messages is worth having. The one I use is built on shared memory that the rank processes map directly, capped at 64 KiB, handing anything larger (prefill, mostly) back to the standard collective. Disabling it in the saved TP4 A/B run cost 4.41 ms per step.

Because it maps shared-memory handles across those processes, it needs SYS_PTRACE and seccomp=unconfined in my container. Those permissions widen what the container can do. They are required for this optional collective, not for running Qwen3.8 in general, so remove them and disable the path if that tradeoff does not suit you.

Hybrid models have another memory pool to watch. The recurrent state pool competes with the KV cache and can bound concurrency before the ordinary request limit does. I once configured it below the number of slots a single request needs and got a server reporting zero runnable requests, with an error that mostly talked about memory. Raising the request limit was never going to help. There was nowhere to put the recurrent state.

The trace blamed the wrong component

At one point I fit TP2 and TP4 measurements to a simple model with a sharded weight term and a constant term. The fit suggested roughly 12 ms that did not shrink with more cards. Host overhead or graph replay looked guilty.

Rather than apportion a busy trace, I priced components by removal on the real captured graph. Replace one component with shape-matched zeros, keep the rest of the graph and the collectives intact, and measure against an unchanged repeat. This does not produce a perfect additive profile, since collectives overlap some module rows, but it answers the causal question: what disappears if this component is removed?

Starting from a 28.59 ms B65 TP2 run:

The supposedly fixed host term was mostly dense MLP device work, whose poor multi-card scaling made a two-point fit look constant. The DeltaNet recurrence itself was small. Had I trusted the fit, I would probably have spent another week optimizing the wrong layer family.

That changed the order of future work: merged four-bit projection shapes first, then dense scaling, then whatever a fresh removal budget turns up. The recurrence is not free, but it is not the first problem.

The final latency distribution is pleasantly boring

A mean TPOT can hide an ugly tail, so I measured the inter-token latency distribution for the final AWQ release. Each point below summarizes 1,023 inter-token intervals from the second run of a 1,024-token output, concurrency one, graph capture on.

Figure 2. Across the short and 8K prompts, p99 stayed within 0.66 ms of the median on TP2 and within 0.57 ms on TP4.

This is the kind of graph I like: not dramatic, just steady. It supports a more useful statement than “the model does 52 tok/s”. At these two contexts and this concurrency, the ordinary token-to-token experience hides no large tail.

What might B70 do?

I have not measured Qwen3.8-27B on B70, so an exact B70 bar would be fiction. There are two useful boundaries though.

The conservative edge assumes no decode uplift at all, since B65 and B70 share the same advertised 608 GB/s of memory bandwidth. The optimistic edge borrows the largest B70-over-B65 decode ratio I measured on two other models, 1.273x. That is deliberately rough: Qwen3.8 is dense while those models are sparse hybrids, so their scaling does not automatically transfer.

Figure 3. The B70 range is a planning aid, not a benchmark. The lower edge assumes equal bandwidth means equal decode; the upper edge applies the largest uplift seen on two earlier models.

Prefill has a better reason to move, since the B70 has more Xe2 cores and roughly 1.86x the advertised dense INT8 compute. Even there I would not simply multiply the measured result by 1.86. Attention, dequantization, collectives, and achieved occupancy do not all scale with peak matrix throughput. The next graph I put here should be a measured one.

A quick detour through the other models

Qwen3.8 was the fourth model in this line of work. The earlier three are not apples-to-apples competitors, since they use different architectures, card counts, TP sizes, and workloads. Their value here is showing which methods survived contact with another model.

The most useful warning came from Ornith. It looked similar to Qwen: hybrid attention, Gated DeltaNet, packed four-bit experts. Yet its asymmetric zero points changed the correctness contract, its router already used a good XPU top-k path, and the Qwen attention override made it slower. The method transferred; the configuration did not.

Those rows are reference points, not a competition between models. Each one deserves its own post, and I plan to write them. This one is about Qwen3.8 and the method I used to work out what it needed.

That is also why I do not present a bag of “Xe2 optimizations” to enable all at once. Graph capture, rooflines, proof that a fast path actually ran, and controlled A/Bs are methods. A tile size, context cutoff, or packed layout belongs to the exact model, card, and TP combination that earned it.

What were Codex and Claude actually useful for?

I used coding agents extensively during this work, mainly Codex and Claude. They were useful in a narrower and more practical way than “the agents wrote the optimization”.

• Reading an unreasonable amount of source. One investigation followed the SYCL grouped GEMM through its tile policy and reorder atoms, then compared it against oneDNN. That is how I learned the clever four-bit conversion I was about to implement already existed.
• Surveying an API surface. An agent enumerated quantized primitives with real XPU dispatch, their schemas, and their layout contracts, which quickly separated usable operators from names that merely sounded relevant.
• Building harnesses from a strict specification. The speculative-decoding harness refuses failed requests and keeps cache-sensitive natural-text tests separate from random-token kernel isolation.
• Running serialized campaigns. Tuning many shapes is boring but valuable, and it is a good background job when only one process may own the GPUs.

They were also entirely capable of producing a confident conclusion from an empty log. Twice a container had stopped and the missing output was treated as a finding. A pkill -f pattern matched the replacement command itself four times. One benchmark reported a rejected over-context request as zeros, which looked like an amazing performance result for a few seconds.

The fix belonged in the harness rather than the prompting: fail loudly, record the exact setup and run number, keep retractions beside the results, and refuse to print metrics for a rejected request. My own job stayed the same: one benchmark on the GPUs at a time, one changed variable, and no number without a saved log.

Fast gibberish is still gibberish

Two quantization mistakes produced clean server starts and output consisting almost entirely of exclamation marks.

First, the text-only model class changed module prefixes. Weight loading had a name-translation layer, but the quantization ignore list did not, so exclusions silently missed their targets.

Then the quantizer added the container module layers.N.linear_attn to the ignore list, because the container itself was not a Linear. The runtime matched ignore entries by substring, so that one entry hid every projection beneath it. Packed tensors existed in the checkpoint and were never loaded into those linears.

Both cases now fail at build time, and I also compare packed dequantization against an independent float64 reference on real tensors. Shape checks alone are not enough. A wrong packed orientation can be shape-valid and still produce fluent-looking, numerically wrong output.

The final group-128 AWQ checkpoint passed health, determinism, and coherence checks on code, reasoning, factual, and summarization prompts. Against the BF16 reference:

Exact argmax agreement applies only where the sequences are still aligned. Every greedy sequence eventually diverged, leaving 7.4% of steps directly comparable. So the claim stays modest: the quantized model tracks BF16 closely before divergence, at a measured +0.46 prefill-perplexity cost. It is still a four-bit model, not an exact BF16 drop-in.

I also have an absolute IFEval result, but no current BF16 run through the same harness and scored subset, and I will not compare it against an older result that skipped a different set of instruction types.

Multimodal inference works. The checkpoint keeps the vision tower in BF16, images load, and the model answers questions about them. What I have not done is score it against a broad image benchmark suite, so I can say the capability is present and working without yet quantifying what the four-bit text weights cost it. The MTP path is enabled as well, though acceptance rates and end-to-end benefit on natural coding prompts still need a proper campaign. Random token IDs are useful for defeating the prefix cache, but they pin speculative yield near its minimum and cannot answer whether MTP helps a real session.

What still does not make sense?

I cut 35% of the bytes read per decode step and got 9% more throughput back. Three independent views agree that the shortfall is real: checkpoint-aware projection, same-shape kernel rates, and component removal. None of them explains why the merged DeltaNet projection behaves so differently from the dense FFN path.

Somewhere in there is performance I have already paid for and am not collecting, and until I know why, every other number in my AWQ build rests on something I cannot account for. So it goes first.

Some of the things I want to do next:

1. Work out where the rest of that saving went. Profile the merged DeltaNet four-bit shapes inside the running server, not as standalone GEMMs, and find out what makes that shape different from the dense FFN projections.
2. Redo the component-removal measurements once that changes, because the rankings expire the moment the algorithm does.
3. Measure Qwen3.8 on B70 and put real numbers where the estimated range is now.
4. See how often MTP’s speculated tokens actually get accepted in real coding-agent sessions instead of synthetic prompts.
5. Run a full multimodal suite so the vision path has numbers too.

What I would carry to the next model is the method rather than this configuration: check upstream first, count bytes from the checkpoint, keep the vendor library as the baseline, verify instruction and layout legality before designing a tile, prove every fast path actually served, and measure with graph capture on, the way the model will be used day to day.

That process is slower than collecting one exciting number, but it made the next model much faster to understand. And Qwen3.8 is now a local coding-agent backend I can use without thinking about every token it generates, which is what I wanted from it.

Cloudflare’s TryCloudflare

Cloudflare’s TryCloudflare lets you quickly expose local services to the internet with no Cloudflare account required. While it’s intended for testing or temporary use, Cloudflare also offers premium tunnels for production-grade needs.

Gradio’s Share Feature

Gradio is widely used for creating interfaces for machine learning models, but its share=True option can also be repurposed to expose general-purpose applications.

Recommendations and When to Use

• Cloudflare TryCloudflare: Best for quick exposure of services without additional Python dependencies. Use it for secure sharing during hackathons or short-lived projects. For production or long-term use, consider Cloudflare’s premium tunnels.

• Gradio Share Feature: Ideal for Python developers or when you need a simple interface to accompany your service. Use it to share ML demos, Python apps, or lightweight experiments.

Both tools are great for development and testing but remember their limitations for production-grade usage. Choose based on your workflow and requirements.

This guide equips you with two powerful tools to expose local applications to the world, simplifying collaboration and testing. Happy sharing!

The Advent of LoRA: A Paradigm Shift in LLM Fine-Tuning

LoRA represents a significant advancement in the fine-tuning of LLMs. By introducing trainable adapter modules between the layers of a large pre-trained model, LoRA focuses on refining a smaller subset of model parameters. These adapters are low-rank matrices, significantly reducing the computational burden and preserving the valuable pre-trained knowledge embedded within LLMs. The key aspects of LoRA include:

• Low-Rank matrix structure: Shaped as (r x d), where ‘r’ is a small rank hyperparameter and ‘d’ is the hidden dimension size. This structure ensures fewer trainable parameters.
• Factorization: The adapter matrix is factorized into two smaller matrices, enhancing the model’s function adaptability with fewer parameters.
• Scalability and adaptability: LoRA balances the model’s learning capacity and generalizability by scaling adapters with a parameter α and incorporating dropout for regularization.

Quantized LoRA (QLoRA): Efficient Fine-Tuning on Intel Hardware

QLoRA advances LoRA by introducing weight quantization, further reducing memory usage. This approach enables the fine-tuning of large models, such as the 70B-parameter Llama 2, on a single GPU , like Intel® Data Center GPU Max Series 1100 with 48 GB VRAM, which was considered impossible previously. QLoRA’s main features include:

• Memory efficiency: Through weight quantization, QLoRA substantially reduces the model’s memory footprint, crucial for handling large LLMs.
• On-the-fly dequantization: It temporarily dequantizes the quantized weights for computations, focusing only on adapter gradients during training.

from bigdl.llm.transformers import AutoModelForCausalLM 

# Loading the model in a 4-bit format for efficient memory usage 
model = AutoModelForCausalLM.from_pretrained( 
    "model_id",  # Replace with your model ID 
    load_in_low_bit="nf4", 
    optimize_model=False, 
    torch_dtype=torch.float16, 
    modules_to_not_convert=["lm_head"], 
) 

Learning Rate and Stability in Training

Selecting an optimal learning rate is critical in QLoRA fine-tuning to balance training stability and convergence speed. This decision is vital for effective fine-tuning outcomes as a higher learning rate can lead to instabilities with the training loss abnormally drop zero after few steps.

from transformers import TrainingArguments 

# Configuration for training 
training_args = TrainingArguments( 
    learning_rate=2e-5,  # Optimal starting point; adjust as needed 
    per_device_train_batch_size=4, 
    max_steps=200, 
    # Additional parameters... 
) 

During the fine-tuning process, there is a notable rapid decrease in the loss after just a few steps, which then gradually levels off, reaching a value near 0.6 at approximately 300 steps as seen in the graph below:

Text-to-SQL Conversion: Prompt Engineering

With the fine-tuned model, we can convert natural language queries into SQL commands, a vital capability in data analytics and business intelligence. To fine-tune the model, we must carefully convert the data into a structured prompt like below to form an instruction dataset with Input, Context and Response fields:

# Function to generate structured prompts for Text-to-SQL tasks 
def generate_prompt_sql(input_question, context, output=""):
    return f"""You are a powerful text-to-SQL model. Your job is to answer questions about a database. You are given a question and context regarding one or more tables. 
You must output the SQL query that answers the question.
### Input:
{input_question}
### Context:
{context}
### Response:
{output}"""

Diverse Model Options

The notebook supports an array of models, each offering unique capabilities for different fine-tuning objectives:

• NousResearch/Nous-Hermes-Llama-2–7b
• NousResearch/Llama-2–7b-chat-hf
• NousResearch/Llama-2–13b-hf
• NousResearch/CodeLlama-7b-hf
• Phind/Phind-CodeLlama-34B-v2
• openlm-research/open_llama_3b_v2
• openlm-research/open_llama_13b
• HuggingFaceH4/zephyr-7b-beta

Model Inference with QLoRA: A Comparative Approach

The true test of any fine-tuning process lies in its inference capabilities. In the case of the implementation, the inference stage not only demonstrates the model’s proficiency in task-specific applications but also allows for a comparative analysis between the base and the fine-tuned models. This comparison sheds light on the effectiveness of the LoRA adapters in enhancing the model’s performance for specific tasks.

from bigdl.llm.transformers import AutoModelForCausalLM 

# Loading the model for inference 
model_for_inference = AutoModelForCausalLM.from_pretrained( 
    "finetuned_model_path",  # Path to the fine-tuned model
    load_in_4bit=True,  # 4 bit loading
    optimize_model=True,
    use_cache=True,
    torch_dtype=torch.float16, 
    modules_to_not_convert=["lm_head"], 
)

Running Inference: Comparing Base vs Fine-Tuned Model

Once the model is loaded, we can perform inference to generate SQL queries from natural language inputs. This process can be conducted on both the base model and the fine-tuned model, allowing you to directly compare the outcomes and assess the improvements brought about by fine-tuning with QLoRA:

# Generating a SQL query from a text prompt 
text_prompt = generate_sql_prompt(…)
# Base Model Inference 
base_model_sql = base_model.generate(text_prompt) 
print("Base Model SQL:", base_model_sql) 
# Fine-Tuned Model Inference 
finetuned_model_sql = finetuned_model.generate(text_prompt) 
print("Fine-Tuned Model SQL:", finetuned_model_sql)

Following a 15-minute session training itself, the fine-tuned model demonstrates enhanced proficiency in generating SQL queries that reflect the given questions more accurately than the base model. With additional training steps, we can anticipate further improvements in the model’s response accuracy:

Finetuned Model:

Base Model:

Check out the notebook on Intel Developer Cloud

I invite AI practitioners and developers to explore the full notebook on the Intel Developer Cloud, where you can experiment with and explore the capabilities of fine-tuning LLMs using QLoRA on Intel hardware with Intel AI software optimizations. Once you log into Intel Developer Cloud, go to the “Training Catalog”. Under “Gen AI Essentials” in the catalog, you can find the LLM fine-tuning notebook and other notebooks.

Repo

You can find the full code and other related notebooks here.

A quick recap / updated steps to set up Arc on Linux

Intel has now published documentation on how to set up Arc on Linux. I tried it today, it worked beautifully.

Stable Diffusion

Stable Diffusion is a fully open-source (thank you Stability.ai) deep learning text to image and image to image model. For more information on the model, checkout the wikipedia entry for the same.

~ → conda create -n ipex python=3.9 -y

~ → conda activate ipex
~ → pip install ~/Downloads/*.whl

~ → pip install diffusers ftfy transformers Pillow

import intel_extension_for_pytorch
import torch
from diffusers import StableDiffusionPipeline

model_id="runwayml/stable-diffusion-v1-5"
prompt = "vivid red hot air ballons over paris in the evening"
pipe = StableDiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,  # this can be torch.float32 as well
    revision="fp16",
    use_auth_token="<the token you generated>")
pipe = pipe.to("xpu")
image = pipe(prompt).images[0]
image.save(f"{prompt[:5]}.png")

In [8]: image = pipe(prompt).images[0]
   ...: 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:35<00:00,  1.43it/s]
In [9]: image = pipe(prompt).images[0]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:09<00:00,  5.20it/s]

~ → pip install git+https://github.com/divamgupta/stable-diffusion-tensorflow ftfy pillow tqdm regex tensorflow-addons

import intel_extension_for_tensorflow
import tensorflow
from stable_diffusion_tf.stable_diffusion import StableDiffusion
from PIL import Image

prompt = "vivid red hot air ballons over paris in the evening"
generator = StableDiffusion(
    img_height=512,
    img_width=512,
    jit_compile=False,
)

img = generator.generate(
    prompt,
    num_steps=50,
    unconditional_guidance_scale=7.5,
    temperature=1,
    batch_size=1,
)
Image.fromarray(img[0]).save("sd_tf_fp32.png")

2022-11-06 23:00:51.948547: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
  0   1: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:00<00:00,  1.21s/it]
2022-11-06 23:01:55.103111: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type XPU is enabled.
  0   1: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:29<00:00,  1.67it/s]

Repo

You can find the full code and other related materials here.

Configure Linux kernel

1. Install the latest available Linux kernel. For example, I have installed kernel 6.0

Install a recent version of the kernel. The easiest way to install a different kernel is by using Mainline.

Coding → add-apt-repository ppa:cappelikan/ppa
Coding → apt update
Coding → apt install mainline

As seen in the screenshot below, I am using Kernel 6.0.0:

After installing the kernel, please restart the machine.

2. As Arc dGPUs support are still experimental in the kernel, you will have to force the dGPU to be detected. This can be done using the kernel force_probe parameter i915.force_probe=<device_id> for the Intel i915 HD graphics driver. This will force probe the driver for new Intel graphics devices that are recognized by the kernel but not adequately supported. Hopefully, you wouldn’t have to do that with a newer version of the kernel. I am using a Yoga 7i (16” Intel) with Intel Arc Graphics which has the Intel® Arc™ A370M discrete Graphics card (dgpu) along with an integrated Intel® UHD Graphics. The device id for this dgpu is 5693, and you can find the device id of the card either by looking at i915 logs using sudo dmesg | grep -i i915 or from Intel’s gpu hardware table.

Pop!_OS 22.04 LTS uses systemd to manage kernel boot params, to force i915 driver to enable the dgpu use kernelstub tool:

Coding → kernelstub -a "i915.force_probe=5693"

After this restart the machine and check i915 logs using dmesg to see if the graphics card has been detected:

Coding → dmesg | grep -i i915

You should see an output like this:

Coding →  dmesg | grep -i i915
[    0.000000] Command line: initrd=\EFI\Pop_OS-97fe6a26-7d8a-4120-89db-8f2130b644b7\initrd.img root=UUID=97fe6a26-7d8a-4120-89db-8f2130b644b7 ro quiet loglevel=0 systemd.show_status=false splash i915.force_probe=5693
[    0.047217] Kernel command line: initrd=\EFI\Pop_OS-97fe6a26-7d8a-4120-89db-8f2130b644b7\initrd.img root=UUID=97fe6a26-7d8a-4120-89db-8f2130b644b7 ro quiet loglevel=0 systemd.show_status=false splash i915.force_probe=5693
[    1.775328] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    1.775383] i915 0000:00:02.0: vgaarb: deactivate vga console
[    1.775416] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    1.775979] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    1.777234] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc_ver2_16.bin (v2.16)
[    1.912397] i915 0000:00:02.0: [drm] GuC firmware i915/adlp_guc_70.1.1.bin version 70.1
[    1.912399] i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9
[    1.926153] i915 0000:00:02.0: [drm] HuC authenticated
[    1.926448] i915 0000:00:02.0: [drm] GuC submission enabled
[    1.926449] i915 0000:00:02.0: [drm] GuC SLPC enabled
[    1.927353] i915 0000:00:02.0: [drm] GuC RC: enabled
[    1.930736] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    3.769893] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    3.775461] i915 0000:03:00.0: enabling device (0000 -> 0002)
[    3.775491] i915 0000:03:00.0: [drm] Incompatible option enable_guc=3 - HuC is not supported!
[    3.776346] i915 0000:03:00.0: [drm] VT-d active for gfx access
[    3.776472] i915 0000:03:00.0: [drm] Local memory IO size: 0x00000003fa000000
[    3.776476] i915 0000:03:00.0: [drm] Local memory available: 0x00000003fa000000
[    3.787361] fbcon: i915drmfb (fb0) is primary device
[    3.787366] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
[    3.796583] i915 0000:03:00.0: [drm] Finished loading DMC firmware i915/dg2_dmc_ver2_06.bin (v2.6)
[    5.326692] i915 0000:03:00.0: [drm] failed to retrieve link info, disabling eDP
[    5.428126] i915 0000:03:00.0: [drm] GuC firmware i915/dg2_guc_70.1.2.bin version 70.1
[    5.442889] i915 0000:03:00.0: [drm] GuC submission enabled
[    5.442892] i915 0000:03:00.0: [drm] GuC SLPC enabled
[    5.443378] i915 0000:03:00.0: [drm] GuC RC: enabled

The dg2_guc is loaded as seen in the above line: [ 5.428126] i915 0000:03:00.0: [drm] GuC firmware i915/dg2_guc_70.1.2.bin version 70.1. Both the igpu and dgpu is detected and also the firmware is loaded. If drm verification fails and GuC is not loaded, install the latest linux firmware from this link:

You can install the latest firmware files by cloning the repo and moving it to /lib/firmware (warning: this is a brute-force approach):

Coding →  git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/
Coding →  cd linux-firmware && yes | cp -r * /lib/firmware

For me, the firmware that came with Pop!_OS 22.04 LTS worked, i didn’t have to install the latest ones.

Setup Open source Mesa 3d Graphics libraries for OpenGL and Vulkan

1. Install drivers

Now, if you want media and graphics support beyond compute, install bleeding edge Mesa libraries from oiabf ppa that provides open graphics drivers; you can do it by:

Coding → add-apt-repository ppa:oibaf/graphics-drivers
Coding → apt-update
Coding → apt-upgrade

Coding → dpkg -l | grep -i mesa
ii  libegl-mesa0:amd64                                                      22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        free implementation of the EGL API -- Mesa vendor library
ii  libgl1-mesa-dri:amd64                                                   22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        free implementation of the OpenGL API -- DRI modules
ii  libglapi-mesa:amd64                                                     22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        free implementation of the GL API -- shared library
ii  libglu1-mesa:amd64                                                      9.0.2-1                                                           amd64        Mesa OpenGL utility library (GLU)
ii  lib
-mesa0:amd64                                                      22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        free implementation of the OpenGL API -- GLX vendor library
ii  mesa-utils                                                              8.4.0-1ubuntu1                                                    amd64        Miscellaneous Mesa utilities -- symlinks
ii  mesa-utils-bin:amd64                                                    8.4.0-1ubuntu1                                                    amd64        Miscellaneous Mesa utilities -- native applications
ii  mesa-va-drivers:amd64                                                   22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        Mesa VA-API video acceleration drivers
ii  mesa-vdpau-drivers:amd64                                                22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        Mesa VDPAU video acceleration drivers
ii  mesa-vulkan-drivers:amd64                                               22.3~git2210120600.ddc5c3~oibaf~j                                 amd64        Mesa Vulkan graphics drivers

Fair warning, be careful as these drivers are bleeding edge and, by design, may not be stable. After installing the Mesa drivers, you should be able to run glx and Vulkan benchmarks.

2. Check if opengl detects the dgpu using glxinfo

Install glxinfo from apt repo mesa-utils

To use the dGPU, set the env variable DRI_PRIME=1, PRIME is a technology in Linux that uses open source graphics drivers to use switchable graphics and install glxinfo from apt repo mesa-utils.

Here is the output of glxinfo without setting DRI_PRIME environment variable:

Coding → glxinfo -B | grep -i device
    Device: Mesa Intel(R) Graphics (ADL GT2) (0x46a6)

After setting the environment variable for PRIME:

Coding → export DRI_PRIME=1
Coding → glxinfo -B | grep -i device
    Device: Mesa Intel(R) Arc(tm) A370M Graphics (DG2) (0x5693)

As you can the dgpu is recognized by Mesa OpenGL. yay!

3. Let’s try to run a benchmark on the Arc gpu to see how it performs. I am using glmark2, which can be installed on Ubuntu-based OSes easily using:

Coding → apt-get install glmark2

Coding → export DRI_PRIME=1
Coding → glxinfo -B | grep -i device
    Device: Mesa Intel(R) Arc(tm) A370M Graphics (DG2) (0x5693)
Coding → glmark2

4. Now, let’s install the intel compute drivers. Goto. Get the latest release and install using the deb packages for OpenCL, level zero, igc etc.

As of this writing, the latest compute driver release version is 22.39.24347:

Coding → cd /tmp && mkdir compute_drivers
Coding → cd compute_drivers
Coding → wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.12149.1/intel-igc-core_1.0.12149.1_amd64.deb
Coding → wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.12149.1/intel-igc-opencl_1.0.12149.1_amd64.deb
Coding → wget https://github.com/intel/compute-runtime/releases/download/22.39.24347/intel-level-zero-gpu_1.3.24347_amd64.deb
Coding → wget https://github.com/intel/compute-runtime/releases/download/22.39.24347/intel-opencl-icd_22.39.24347_amd64.deb
Coding → wget https://github.com/intel/compute-runtime/releases/download/22.39.24347/libigdgmm12_22.2.0_amd64.deb
Coding → dpkg -i *.deb

Now that the compute drivers are installed let’s see if OpenCL can detect the dgpu. Install clinfo from apt and check using:

Coding → clinfo | grep "0x5690"
  Device Name                                     Intel(R) Graphics [0x5693]
    Device Name                                   Intel(R) Graphics [0x5693]
    Device Name                                   Intel(R) Graphics [0x5693]
    Device Name                                   Intel(R) Graphics [0x5693]

We can see that the dgpu has been detected.

Install oneAPI basekit and device discovery using sycl

1. Finally, install the oneapi basekit to use the dpcpp runtime. I used 2022.2.0 version of oneapi basekit.

Please refer to oneAPI installation guide to install using the apt package manager.

Source the oneAPI environment using:

→ source /opt/intel/oneapi/setvars.sh

~ → dpcpp -v
Intel(R) oneAPI DPC++/C++ Compiler 2022.2.0 (2022.2.0.20220730)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/intel/oneapi/compiler/2022.2.0/linux/bin-llvm
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

2. Device discovery using sycl-ls see if sycl can detect the dgpu:

Intel dgpus like the A370m are represented as SYCL devices. sycl-ls is a tool that is part of the oneAPI basekit that can show all the detected devices and all the SYCL backends support by the runtime. Once the oneapi basekit has been installed, source the environment using:

Coding → source /opt/intel/oneapi/setvars.sh 

Device discovery using syclto see if sycl can detect the dgpu:

Coding → sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.14.7.0.30_160000]
[opencl:cpu:1] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i7-12700H 3.0 [2022.14.7.0.30_160000]
[opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) Graphics [0x5693] 3.0 [22.40.024349]
[opencl:gpu:3] Intel(R) OpenCL HD Graphics, Intel(R) Graphics [0x46a6] 3.0 [22.40.024349]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Graphics [0x5693] 1.3 [1.3.24349]
[ext_oneapi_level_zero:gpu:1] Intel(R) Level-Zero, Intel(R) Graphics [0x46a6] 1.3 [1.3.24349]

To get a more verbose output, use, sycl-ls --verbose:

Coding → sycl-ls --verbose | grep -i name
    Name     : Intel(R) FPGA Emulation Platform for OpenCL(TM)
        Name       : Intel(R) FPGA Emulation Device
    Name     : Intel(R) OpenCL
        Name       : 12th Gen Intel(R) Core(TM) i7-12700H
    Name     : Intel(R) OpenCL HD Graphics
        Name       : Intel(R) Graphics [0x5693]
    Name     : Intel(R) OpenCL HD Graphics
        Name       : Intel(R) Graphics [0x46a6]
    Name     : Intel(R) Level-Zero
        Name       : Intel(R) Graphics [0x5693]
        Name       : Intel(R) Graphics [0x46a6]
    Name     : SYCL host platform
        Name       : SYCL host device

As seen above 2 GPU devices are detected by the SYCL runtime and are supported using both OpenCL and Level-Zero drivers.

We now have configured the machine with all the required software stack to fully utilize the discrete gpu available on the laptop. These instructions can be used to enable any Arc discrete GPUs like the A370m, A770m, A770, A750 etc on Linux.

Intel’s System Monitoring Utility to monitor the dgpu

1. Install sysmon

sysmon is a tool similar to top for cpu, that is part of Intel’s Platform Tools Interfaces for GPU. sysmon helps in monitoring the Intel gpu parameters like frequency, memory, etc. The tool can be installed using:

Coding → git clone https://github.com/intel/pti-gpu/
Coding → cd pti-gpu/tools/sysmon
Coding → mkdir build && cd build
Coding → cmake -DCMAKE_BUILD_TYPE=Release .. && make

2. After successfully building sysmon let’s check the dgpu frequency:

./sysmon

As seen above both the igpu and dgpu performance can be monitored using the tool.

Acknowledgements

I wouldn’t have been able to do this without the help of @sanchitintel and @gujingui. Also, Phoronix has been publishing updates on the best way to enable Intel Arc dGPU, detailing the version of the kernel, mesa drivers, etc. That was my start in setting up the software stack.

Something not working?

Please create an issue here to track any issues with these steps.

Citation

If you are using this information, please cite using the below link:

Unnikrishnan Nair, R. (2022). dgpu_setup_pytorch (Version 1.0.0) [Computer software]. https://github.com/rahulunair/intel_arc_dgpu_linux

What is it ?

use std::any::type_name;

// what is the type of the passed argument
fn what_type<T>(_: &T) {
    println!("type is: {}", type_name::<T>());

}

type_name function from the std crate returns the name of a type as a string slice.

Before knowing about this function available from the std crate(reminder to self, to read the standard library docs!), i tended to use the ide introspection features to confirm what the type of a variable was at times or use a () as type and the compiler complains to me that the type is something else, which is what i wanted to known in the first place, but know I have a shiny new function that I can use, lets see how it works.

How is it used ?

Let’s take a simple example,

fn main() {
    let number = 3232;
    let name = "Rahul".to_string();
    let list_of_nums = vec![1, 2, ];
    what_type(&number);
    what_type(&name);
    what_type(&list_of_nums);
}

That is it, and it will print the type of the variable passed into the what_type function as below:

type is: i32
type is: alloc::string::String
type is: alloc::vec::Vec<i32>

References

type_name

End

Like I said a short post about a really useful(hopefully) function in Rust.

What is it ?

It is to specify a concrete type, for a function, a struct, a method or an enum. The syntax of turbofish looks like:

::<T>

It looks kinda like a fish I guess, the name turbofish, check steve klabnik’s official explanation here for more details.

How is it used ?

Let’s take a simple example, to parse a string into float,

fn main() {
    let pi_string = "3.1415";
    let pi_float = pi_string.parse().unwrap();
    println!("{}", pi_float);
}

This program will not compile as parse function is too generic, and the rust compiler being helpful as always tell us to give a time annotation to the variable pi_float:

 Compiling turbofish v0.1.0 (/Users/unrahul/Coding/rust/learn/turbofish)
error[E0282]: type annotations needed
 --> src/main.rs:3:9
  |
3 |     let pi_float = pi_string.parse().unwrap();
  |         ^^^^^^^^ consider giving `pi_float` a type

This can be solved in two ways, either we use a turbofish notation for parse function or use type annotation, let’s see how we can use turbofish to solve this.

fn main() {
    let pi_string = "3.1415";
    let pi_float = pi_string.parse::<f32>().unwrap();
    println!("{}", pi_float);
}

We tell the compile, which concrete type is being parsed using parse, and the code compiles:

Compiling turbofish v0.1.0 (/Users/unrahul/Coding/rust/learn/turbofish)
Finished dev [unoptimized + debuginfo] target(s) in 0.21s
Running `target/debug/turbofish`
3.1415

Lastly, let’s try the compiler suggested fix, that is to use type annotation:

fn main() {
    let pi_string = "3.1415";
    let pi_float : f32 = pi_string.parse().unwrap();
    println!("{}", pi_float);
}

Yup, that works too!, as shown below:

Compiling turbofish v0.1.0 (/Users/unrahul/Coding/rust/learn/turbofish)
Finished dev [unoptimized + debuginfo] target(s) in 0.21s
Running `target/debug/turbofish`
3.1415

References

Where to put turbofish
What is Rust’s turbofish

End

This was a quick introduction to the turbofish notation and how it is used.

References

For more details, please read:

Rust docs on Result type
Unwrap and Expect in Rust
Unrwap and Expect

The end

In this post, we got a basic idea of what Results are and how to handle them. Next time, we can see how to handle errors in a bit more detail.

Simple methods to try first:

• Regularize with dropout and maxout
• Store weights for each environment
• Take average of these weights and initialize when learning a new environment
• Add adaptive learning rate for weights, slow learning rate for weights for common skills for each environment and fast learning rate for new environments

Ideas to further investigate:

• PathNet - For learning similar tasks
• Elastic Weight Consolidation - Good for dissimilar tasks
• H-DRLN - Hierarchal Deep Life Long RL – - A framework based that combines DQN and lifelong learning techniques

Before we dive in, lets see why does a neural network ever have ‘Catastrphic Forgetting’ problem?

Catastrophic Forgetting is not only limited to neural nets; if that was the case, we could have easily replaced it some other learning algorithm. Researchers who studied this problem were of the opinion that the underlying cause is the generalization that a neural net does. The ability of a neural net to distribute and share representation of the relationship between input and an output label helps it to generalize, which is one of the most important properties of a neural net. Thus, each time a new data point (x, y) comes up to be trained, the neural net tries to distribute and share the representation by adjusting the weights of the neural net. This leads to the neural net forgetting earlier representations.

What are some of the possible solutions?

Many methods that detailed below can be explained off as a particular type of regularization. The techniques given below can be considered as a bag of tricks that could be used to limit catastrophic forgetting. As mentioned above, most of these solutions can be explained as a type of regularization and freezing of learned structures. To remember everything is to not change any of the weights or the activation traces of a task. Some methods directly tackle this by freezing weights, others by freezing activation traces of the entire network from input to output.

Regularization, dropout and activation functions

• Regularization
• Joint Many-Task model - Successive regularization at each epoch
• Dropout with Maxout

Regularization is traditionally used to prevent overfitting; this can be done in many ways. A standard way to do this is to add a norm component to the loss function that is proportional to the square of the weights of a neural network (L2 norm). L2 regularization has the effect of the network preferring to learn smaller weights; as more significant the weight, larger would be the loss value. How this prevents forgetting can be attributed to the fact that, when the weights are small (regularized), small input values will not affect the loss function drastically. This, in turn means, the gradient of the loss would be a smaller value, thus when the weights which are adjusted based on this gradient value will not change much.

Dropout is used as a regularizing technique. Here how it regularizes can be explained by considering dropout as training a set of neural networks (each time dropping out a set of hidden units), then averaging the result of an ensemble of nets at the end [dropout]. This makes the model robust to losses by not assuming that a set of neurons (thus information) will always be present. Thus, when a new task is learned, the combined effect of a robust and regularized network can to some degree minimize forgetting.

Dropout has been empirically proven to help in adapting to new task while remembering old tasks. It has been suggested to use maxout as the activation function when using dropout as a technique to minimize forgetting.

Weight freezing/slow updating

• Using fast weights to de-blur old memories - Hinton
• Elastic Weight Consolidation - Deepmind

Here the idea is to not frequently update parameters of a Neural Net when learning new tasks. The larger the activation weights between two nodes for a task, the less chance that these weights should be affected. Hinton worked on this problem in the 80s and designed a network with weights of different rate of plasticity [Fast weights]. A similar approach was chosen by Kirkpatrick and the Deepmind team with the Elastic Weight Consolidation (EWC) technique. Here a constraint is added to the loss function that controls which weights can be updated and which cannot. When a new task is being learned, strong activation paths are not updated, and weights that didn’t contribute much to a previous task is updated first.

The insight that led to EWC is that, in a deep neural network, there can be many different configurations of weights that will give us a similar rate of error for a task. The goal with EWC is to find a set of weights from the parameter space that has low error rate for both the new task and the old one. In this approach the authors consider gradient descent in a Bayesian perspective. Whereas stochastic gradient descent tries to estimate a single parameter (weight), they tried to estimate a parameter for the entire distribution of data (Bayesian estimate), now this is not tractable, so what they do is that they use a trick called Laplace approximation and they call their approach EWC. By learning a distribution of parameters for each task, they were able to sample a set of parameters (weights) that worked for both tasks. [EWC].

Ensemble methods

Progressive Neural Networks - Perfect memory PathNet - Deepmind

Ensemble methods attempt to train multiple networks and combine them to get the result, essentially training separate classifiers for each new task.

Progressive neural networks progressively extend its structure in proportion to new tasks. It starts with an initial neural net and to learn a new task; new lateral connections are formed in parallel to the initial one. This helps in few ways, first when a new task has to be learned the previous network weights are frozen thus eliminating catastrophic forgetting and as lateral connections between each layer are formed the new network shares knowledge from the previous network. The only disadvantage with a progressive network is with the growth of new tasks, the network weights also grow.

PathNet is a network of networks. It has a fixed set of networks in which each layer has a collection of parallel mini neural network modules. Optimal paths are discovered using a genetic algorithm or reinforcement learning from the fixed size neural network for each task. After training and identification of a path, they are frozen. Thus the network does not forget. The cool thing about PathNet is that the base network is fixed and it does not grow further than the initial architecture. It is possible for the learned representations to be reused as well. PathNet is considered the best model for this type of learning.

Memory Methods – Rehearsal

• Episodic Memory Approach
• Episodic Generative Approach
• Dual memory models
• Gradient Episodic Memory

These involve both static memory and generative memory methods. Static memory methods involve using a big experience replay type of memory to store previous session data, and during training of each new task, data from memory is randomly sampled and mixed with the new training data. The generative model can be a type of auto-encoder or a generative adversarial network (GAN), where the statistics of the data is stored and replayed while training for new tasks.

Dual memory models use a system of short-term memory (STM) modules and a long-term memory (LTM) module. Both of the memories have a combination of a generator and a learner. The STM which is a collection of task-specific networks, also, has a hash-table that keeps a tab on how many tasks the agent has learned and indexes each of the task-specific networks. The LTM uses a collection of the raw data sample from all the previous tasks to train its generative memory. The STM is used for fast training, and LTM is used to reinforce the STM.

In Gradient Episodic Memory, when learning in a new environment, weight update depends on a meeting a metric. Here for each task, a sample of the states (X-y) are stored in memory. For each update, the gradient is compared with the gradient of all the previous tasks, if the gradient of the current update does not contribute to the overall gradients of all the previous tasks, then an approximation that does contribute is chosen as the update. This reduces drastic changes to weights when learning new tasks, hence prevents catastrophic forgetting.

Now let me give some details on approaches specific to reinforcement learning.

Life Long Reinforcement Learning

A lifelong reinforcement learning agent should have 2 main abilities: Efficiently retain knowledge base of learned policy in an environment (highly regularized sparse data structure). Efficient transfer - Should have the ability to transfer knowledge from the previous environment to the new one.

In life-long reinforcement learning, instead of learning each task, the objective is for the agent to distill learning from different environments, use shared knowledge and learn environment specific information as well. In the following section task and environment mean the same thing.

Life Long RL through multiple environments

These learning methods are based on introducing bias into learning. Here, bias refers to knowledge about the environment. So a lifelong learning agent is initialized with something called the initial bias which is the weights of the previous environment and then updated with learning bias for the new environment. Now, this is in some ways similar to what our DQN agent is doing where we initialize the net with the weights trained from the initial simulator and then use this to bootstrap for the real environment. This can be extended, and initial bias could be the average of weights for all the previous environments. Also, adaptive learning rates are used for weights. The weights are updated in proportion to the amount by which they varied for the previous environments. If they did not vary by a threshold value, the learning rate for those weights are set to be very low, and if the weights beyond a threshold, then that means that these weights are environment dependent, thus the learning rate for these weights for the new environment is set to a higher one.

Most of the recent lifelong learning schemes used in RL is based on an algorithm known as ELLA (Efficient Lifelong Learning Algorithm). The principle behind ELLA is that we assume each task is sequentially submitted to the agent, and each learnable parameter is considered as a linear combination of a latent basis component (a component common to all tasks) and a task-specific component. Thus, the objective would be to minimize predictive loss over all tasks while encouraging significance of the latent shared component. This is done by introducing a sort of inductive bias (inductive of shared task information) to the loss function.

Policy Gradient - ELLA

The goal of policy gradient using efficient lifelong learning algorithm is to find optimum policies parameterized by a set of weights for each environment. We use the same approach as in general ELLA where the parameters (weights) for each task is considered to be a linear combination of parameters of the common latent model between the tasks and task-specific weights. Here the latent structure is stored in memory for future learning, and the PG algorithm in each of its iteration uses this knowledge base to update both latent weights and task-specific weights.

Hierarchical Deep Reinforcement Lifelong Learning Network (H-DRLN)

To solve complex tasks, rather than just knowing what action to take in a state, an agent has to learn skills; these would include things like picking up an object, moving from a point to another point, etc. Reinforcement learning was extended using the options framework to do exactly this. [options framework]. To use reusable skills in a lifelong manner, an algorithm should be able to learn a skill (Eg: how to move left or right); enable an agent to determine which of these skills should be used/reused and cache reusable skills.

H-DRLN has something called a Deep Skill Network which stores independent DQNs that have special skills stored, for example, there could be 2 DQNs as in our case one for passive control and another for active control. Along with the Deep Skill Network, the agent has a generic output layer that outputs either simple actions or skills from the Deep Skill Network depending upon the state sampled. This agent can be considered as a DQN with actions being temporally extended to solve tasks. For training the agent, a modification is made to the experience reply, and a new memory called Skill Experience Reply (S-ER) is used.

With this, we come to the end of the survey on Catastrophic Forgetting and lifelong learning approaches. A few other methods like explicit sparse coding, policy distillation, and curriculum learning have not been mentioned here as many of the techniques discussed could be considered as variations of these algorithms.

Terms

Plasticity - The propensity of a weight to be affected by a change. A weight with high plasticity can be modified easily than a weight with lower plasticity. Auto-encoder - A type of neural networks used to learn representation of data, mainly for dimensionality reduction GAN - GAN or Generative Adversarial Networks uses a competing pair of networks to learn the representation of data and can be used to generate data points from the learned distribution.

References

[Dropout] : https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf [Empirical study] : https://arxiv.org/pdf/1312.6211.pdf [Fast weights]: https://www.cs.bham.ac.uk/~jxb/PUBS/COGSCI05.pdf [EWC]: https://arxiv.org/pdf/1612.00796.pdf [Lifelong ML]: https://www.cs.uic.edu/~liub/lifelong-machine-learning-draft.pdf [Options framework]: http://www-anw.cs.umass.edu/~barto/courses/cs687/Sutton-Precup-Singh-AIJ99.pdf [Multi task learning PG]: http://proceedings.mlr.press/v32/ammar14.pdf [Gradient Episodic Memory]: https://arxiv.org/pdf/1706.08840.pdf [PathNet]: https://arxiv.org/pdf/1701.08734.pdf [Deep Generative Memory]: https://arxiv.org/pdf/1710.10368.pdf [Progressive Neural Net]: 1606.04671 Progressive Neural Networks

The end

Thats all for now folks on Catastrophic Forgetting in Learning systems using connectionist methods.

Temporal Difference or TD Learning

From each step learn something that would make enable us to improve the estimated value for the next step. Consider this, three scenarios, in which the third scenario depends on the second and or the first. If that is the case, then knowing the states in the scenario can help us in better predicting the states in scenario three. We can improve the prediction in the third scenario if there is any change in states for either of the other scenarios, rather than waiting for the third one to finish and then realizing our prediction was close or way off. Consider, you are going somewhere and you expect to go through 2 cities. You estimate that you would reach the destination in 3 hours, as you know or estimate that you will need 1 hour each to cover the 2 other cities. Now, if it’s your lucky day and traffic is low in the first city, thus you could pass through it in 30 minutes instead of 60. Thus, you can estimate that you will reach the final destination 30 minutes early. While passing through the second city, you have car trouble and it takes an hour to fix it and start again. Thus, you now predict you would reach 30 minutes past the estimated time at your destination (Provided, you don’t face any further uncertainties). This continuous improvement of estimate is the main principle behind temporal difference or TD learning. We can say, TD learning is an on-line learning (as we don’t need to wait for the entire episode to finish before updating our estimates). It bootstraps on the estimated value of other states to estimate value of the state in concern.

Types : TD(0) and TD(lambda)

Q learning

It is a variation of TD(0) learning, where we incrementally estimate the Q value for a state based on immediate rewards and the Q value for the next state. The variation is that, to estimate the Q value for the next state, we add the immediate reward with the Q value for the next state that maximizes the value (Q value for the state for the action that gives the maximum value). Also, unlike TD(0) learning, Q learning is an off-policy learning algorithm. Thus the estimated Q value at instance k is, the Q value at k for the state and action at time t plus the difference between estimated Q value using the immediate reward and discounted Q value for the next state for the action that gives the maximum value and the Q value of the current state. This delta between estimated Q value for the next state and Q value for the current state is weighted by a factor called the learning rate, which is between [0, 1]. The weighting factor or learning rate alpha can be decreased based on each iteration or as in many scenarios, use a small fixed value. The rate basically determines by how much we update the Q value. Now, if that sounds complex, trust me, it’s not, I am just not that good at explaining I guess. Just search for the algorithm on line and you will get it instantly.

How is it done?

Well, basically, the agent at time, t, in state, s, does an action, a, and moves to the next state and thus receives a reward, r. Now at time, t+1, the agent knows that it is in state, s, and knows the reward it obtained from the previous state. It uses this information along with the Q value for the optimum action for this state to get a better estimate of the Q value of the state, s. For this update to work, we need to have some starting point of Q values, in practice this is assumed to be zero or set randomly and at each iteration, k, the Q value is updated a little bit based on the learning rate to be a little closer to reality. In time, after many iterations of learning, the Q values for each states will reflect the real values that can be obtained. It has been proved (don’t ask me how, I haven’t checked out the proof) that if we do this iterative update an infinite time, we will eventually get the right Q values, irrespective on the initial Q values, the actions we took in each state etc.

SARSA learning

First of all, what a creative naming, it must have taken them a long time to come up with this name. So, why is it called SARSA?, It’s because, the learning algorithm uses the present state, S, the action taken, A, the reward obtained, R, the next state, S, and the action taken, A, in this next state, while following a policy. SARSA stands for State-Action-Reward-State-Action!. While in traditional Q value function, the objective was to estimate the optimum policy doing exploration of a random policy, in SARSA, we start with a policy and tries to estimate the Q value of starting at a state, doing an action in state and following a policy, that is not changed in the course of learning. Thus it is an on-policy learning algorithm, as we don’t change the policy that has been chosen for a particular iteration. The idea is that if we are able to try all the states and all the possible actions infinitely many times, then this will eventually converge to the optimum policy itself. For a particular iteration, it computationally less demanding that Q learning, but overall, it may need more time to converge. This learning is used when the state transition probabilities might not be fixed, there can be changes in the probabilities of switching from one state to the other. The only change in the algorithm compared to Q learning is, it doesn’t take the action that maximizes the utility when calculating the next state value, but obeys by the policy and takes the stipulated action. Here, too if you don’t understand it, please check the algorithm once on line and it will be clear.

Actor Critic Algorithms

This class of learning algorithms has two parts, the Actor, which is a policy function and the Critic, which is a value function that is used to obtain the value for a state transition following an action. After an action has been selected, the Critic estimates the value for the state and calculates value for the state (immediate reward plus value for the next state). This is used to evaluate the action taken, which is the difference between the calculated value (using Bellman equation) and the estimated value. This difference or delta is used to improve the probability for the action in that state. The improvement factor delta is weighted by a learning rate, beta. A variation of this algorithm called the A3C algorithm is one of the fastest RL algorithms out there.

The end

Awesome, thus in a two step process, we are able to identify an optimum policy based on nothing but a Q value table for each state action pairs. Kind of cool right!

Temporal Difference or TD Learning

From each step learn something that would enable us to improve the estimated value for the next step. Consider this, three scenarios, in which the third scenario depends on the second and or the first. If that is the case, then knowing the states in the scenario can help us in better predicting the states in scenario three. We can improve the prediction in the third scenario if there is any change in states for either of the other scenarios, rather than waiting for the third one to finish and then realizing our prediction was close or way off.

Consider, you are going somewhere and you expect to go through 2 cities. You estimate that you would reach the destination in 3 hours, as you know or estimate that you will need 1 hour each to cover the 2 other cities. Now, if it’s your lucky day and traffic is low in the first city, you could pass through it in 30 minutes instead of 60. Thus, you can estimate that you will reach the final destination 30 minutes early. While passing through the second city, you have car trouble and it takes an hour to fix it and start again. Thus, you now predict you would reach 30 minutes past the estimated time at your destination (Provided, you don’t face any further uncertainties). This continuous improvement of estimate is the main principle behind temporal difference or TD learning.

We can say, TD learning is an on-line learning (as we don’t need to wait for the entire episode to finish before updating our estimates). It bootstraps on the estimated value of other states to estimate value of the state in concern.

Types : TD(0) and TD(lambda)

Q learning

It is a variation of TD(0) learning, where we incrementally estimate the Q value for a state based on immediate rewards and the Q value for the next state. The variation is that, to estimate the Q value for the next state, we add the immediate reward with the Q value for the next state that maximizes the value (Q value for the state for the action that gives the maximum value). Also, unlike TD(0) learning, Q learning is an off-policy learning algorithm. Thus the estimated Q value at instance k is, the Q value at k for the state and action at time t plus the difference between estimated Q value using the immediate reward and discounted Q value for the next state for the action that gives the maximum value and the Q value of the current state. This delta between estimated Q value for the next state and Q value for the current state is weighted by a factor called the learning rate, which is between [0, 1]. The weighting factor or learning rate alpha can be decreased based on each iteration or as in many scenarios, use a small fixed value. The rate basically determines by how much we update the Q value. Now, if that sounds complex, trust me, it’s not, I am just not that good at explaining I guess. Just search for the algorithm online and you will get it instantly.

How is it done?

Well, basically, the agent at time, t, in state, s, does an action, a, and moves to the next state and thus receives a reward, r. Now at time, t+1, the agent knows that it is in state, s, and knows the reward it obtained from the previous state. It uses this information along with the Q value for the optimum action for this state to get a better estimate of the Q value of the state, s. For this update to work, we need to have some starting point of Q values, in practice this is assumed to be zero or set randomly and at each iteration, k, the Q value is updated a little bit based on the learning rate to be a little closer to reality. In time, after many iterations of learning, the Q values for each states will reflect the real values that can be obtained. It has been proved (don’t ask me how, I haven’t checked out the proof) that if we do this iterative update an infinite time, we will eventually get the right Q values, irrespective on the initial Q values, the actions we took in each state etc.

SARSA learning

First of all, what a creative naming, it must have taken them a long time to come up with this name. So, why is it called SARSA?, It’s because, the learning algorithm uses the present state, S, the action taken, A, the reward obtained, R, the next state, S, and the action taken, A, in this next state, while following a policy. SARSA stands for State-Action-Reward-State-Action!. While in traditional Q value function, the objective was to estimate the optimum policy doing exploration of a random policy, in SARSA, we start with a policy and tries to estimate the Q value of starting at a state, doing an action in state and following a policy, that is not changed in the course of learning. Thus it is an on-policy learning algorithm, as we don’t change the policy that has been chosen for a particular iteration. The idea is that if we are able to try all the states and all the possible actions infinitely many times, then this will eventually converge to the optimum policy itself. For a particular iteration, it is computationally less demanding that Q learning, but overall, it may need more time to converge. This learning is used when the state transition probabilities might not be fixed, there can be changes in the probabilities of switching from one state to the other. The only change in the algorithm compared to Q learning is, it doesn’t take the action that maximizes the utility when calculating the next state value, but obeys by the policy and takes the stipulated action. Here, too if you don’t understand it, please check the algorithm once on line and it will be clear.

Actor Critic Algorithms

This class of learning algorithms has two parts, the Actor, which is a policy function and the Critic, which is a value function that is used to obtain the value for a state transition following an action. After an action has been selected, the Critic estimates the value for the state and calculates value for the state (immediate reward plus value for the next state). This is used to evaluate the action taken, which is the difference between the calculated value (using Bellman equation) and the estimated value. This difference or delta is used to improve the probability for the action in that state. The improvement factor delta is weighted by a learning rate, beta. A variation of this algorithm called the A3C algorithm is one of the fastest RL algorithms out there.

The end

Awesome, thus in a two step process, we are able to identify an optimum policy based on nothing but a Q value table for each state action pairs. Kind of cool right!

What is it?

Learning by doing, where the quality of learning is determined by how much reward one can get at the end of an episode.

Some terms that people drop when talking about RL

The end

Awesome, thus in a two step process, we are able to identify an optimum policy based on nothing but a Q value table for each state action pairs. Kind of cool right!

Python multiprocessing

The usual approach I would choose to do this is using the multiprocessing library, although threads or asyncio can be used as well, as this were heavy IO based functions.

Consider these two functions as an example:

def sum():
  a = 0
  for i in range(1000000):
    a += i
  return a

def product():
  a = 0
  for i in range(100000):
    a *= n
  return a

One way to multiprocess sum() and product() would be:

from multiprocessing import Process

p1 = Process(target=sum)
p2 = Process(target=product) 
p1.start()
p2.start()
n_sum, n_prod = p1.join(), p2.join()

That was fairly simple wasn’t it? Although real world functions will have parameters and might have need to share memory etc.. I am avoiding all those here for simplicity.

Using Ray to do the same thing

import ray

# initialize ray
ray.init()

# decorate function that need to be parallized
@ray.remote()
def sum():
  a = 0
  for i in range(1000000):
    a += i
  return a

# decorate function that need to be parallized
@ray.remote()
def product():
  a = 0
  for i in range(100000):
    a *= n
  return a


sum_id, prod_id = sources.remote(), binaries.remote()
# block until finished
n_sum, n_prod = ray.get([sum_id, prod_id])

That’s it, that was as simple as using the multiprocessing library. The interesting thing about ray is that the same code can be used to distribute the workload across a cluster as well without changing the code in any way- this I think is a game changer especially when I don’t have to deal with MPI configs or spark clusters to get that ability.

Running this code, I get a INFO message:

2020-05-23 18:05:18,848 INFO resource_spec.py:204 -- Starting Ray with 7.28 GiB memory available for workers and up to 3.66 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-05-23 18:05:19,246 INFO services.py:1168 -- View the Ray dashboard at localhost:8265

The dashboard running on localhost:8265 gives me a overview of resource consumption *(cpu, memory), amount of data seent and recieved etc, this is especially useful in a distributed cluster environment, very neat!

The end

For distributed Machine Learning training I used Horovod exlusively, the one thing with Horovod is that it needs MPI and at times, it’s a pain to debug MPI, I would like to tryout Ray and see if it is easier to use in place of Horovod. My initial thoughts are all positive and it looks really great.

Installing tiemy

The module is fairly easy to install, you just need python and pip:

pip install tiemy

That’s it!

How to use tiemy

Consider a prime number sieve as below is saved in a file prime.py

from tiemy import timer

@timer
def sundaram(N):
    """sundaram's sieve to find non primes."""
    primes = list(range(3, N, 2))
    half = N // 2
    init = 4
    for step in range(3, N, 2):
        for i in range(init, half, step):
            primes[i-1] = 0
        init += 2 * (step + 1)
        if init > half:
            return [2] + list(filter(0, numbers))


if __name__ == "__main__":
    sundaram(100000)

The output would be something like:

python prime.py
timing func:: sundaram
mean:     0.0090, std:     0.0003 sec.

The above method to find prime numbers is called sundaram sieve ¹, we will look at it more in detail in a later post :).

The end

The program

Fetch n ebooks from project gutenberg site and do some elemental processing on the content.

Some of the code snippets:

def download(url: str) -> str:
    """download and decode content from a url."""
    page = requests.get(url)
    name = url.split("/")[-1]
    return page.content.decode("utf-8"), f"texts/{name}"

def save(content: str, fname: str = None):
    """save content to file with name."""
    with open(fname, "w") as fh:
        fh.write(content)

def process(string: list) -> list:
    """cpu bound tasks"""
    ...
    ...
    return string

def main():
    urls = [
        "https://www.gutenberg.org/cache/epub/376/pg376.txt",
        "https://www.gutenberg.org/files/84/84-0.txt",
        "https://www.gutenberg.org/cache/epub/844/pg844.txt",
    ]

    # call download

    # call save

    #call string process

The full source code is on github at pyjuggle.

Python and concurrency

Let’s get this straight, concurrency and parallelism are not the same. Concurrency means, running multiple programs as though they are running at the same time, where as parallelism is running the programs truly in parallel in multiple cores of your machine.

The interesting aspect of Python (Ruby and few others) is that there is something called the global interpreter lock or GIL. This is part of the cpython implementation, where only one Python object can have access to the interpreter at one time. This means that, we cannot run two threads of execution parallely in Python but that all threads of execution are time sliced. You can read more on GIL here.

Threads come handy though when there is a lot of IO involved or when there are subprocess calls, where the interpreter is handing-off to work to another subsystem, and a new thread of execution can start, while the previous thread waits for a response.

When we need to real multi-core parallel tasks in Python, mulitprocessing library comes to our favour. It essentially spins up multiple Python interpreters and works on orthogal exlusive chunks of data, this method is good for CPU bound tasks.

When to use what?

Although, there are exceptions, I would say when IO or subprocess calls are involved or when the interpreter is giving of control to another sub-system use threads.

The module of choice for me for concurrency in python is concurrent.futures.

To use a thread pool, we can do this:

from concurrent.futures import ThreadPoolExecutor as tpe

    # call download method concurrently
    with tpe(max_workers=3) as exe:
        results = exe.map(download, urls)

Here, maximum of 4 threads will be spun up, each time sliced, and will start downloading the urls. One thing to note here is that, depending upon the amount of data, here the urls threaded code may not show significant performance improvement when compared to one without threading, so it is always good to time sequential and threaded code before you decide.

{% include info.html text=“use threads when the task is IO bound” %}

The same approach can be used to save the files, create a threadpool and save the downloaded content:

from concurrent.futures import ThreadPoolExecutor as tpe
    ...
    ...
    # call save method concurrently
    with tpe(max_workers=3) as exe:
        exe.map(save, results)
    ...
    ...

Finally we come across, a CPU bound task, which has to tokenize the saved files, here if we use threading, although there could be some speedup than serial execution, it is advicable to use multiprocessing, in some cases, threading could actually slow down the program if the program is mostly CPU bound.

An example of using a multiprocess executor is:

from concurrent.futures import ProcessPoolExecutor as ppe

    # call process method parallely
    with ppe(max_workers=4) as exe:
        exe.map(process, open("./all.txt").readlines())

{% include info.html text=“use processes when the task is CPU bound” %}

Here we are mapping 4 process methods with chunks of data from the file, one thing to note here is that the amount of work distributed to the 4 processes is not controlled by the programmer and may not be equal.

The end

This was an interesting experience, I wanted to write about this for few years now, finally got to it. I am pretty sure I have got some things wrong, if you find any such techincal inacurracies ping me on twitter or on github.

May be next time I will write, when to use async-await.

Immutable by default

By default all variables in Rust are immutable, what that means is, once we assign a value to a variable, we cannot reassign another value. The keyword let is used to create variables and bind values to them in Rust.

fn main() {
  let name = "unrahul";
  println!("name is {}", name);

Output of the above program is:

name is unrahul

There you go, what we have done is; we have created a string literal name in Rust of type &str. Now, if we try to reassign the same variable name in the same scope as main:

fn main() {
    let name = "unrahul";    // created a variable `name` and assigned a value to it
    println!("name is {}", name);
    name = "rahul";    // trying to assign a new value to the variable `name`  
    println!("new name is {}", name);
}

The compiler will throw an error:

error[E0384]: cannot assign twice to immutable variable `name`
 --> src/main.rs:4:5
  |
2 |     let name = "unrahul";
  |         ----
  |         |
  |         first assignment to `name`
  |         help: make this binding mutable: `mut name`
3 |     println!("name is {}", name);
4 |     name = "rahul";
  |     ^^^^^^^^^^^^^^ cannot assign twice to immutable variable

error: aborting due to previous error
For more information about this error, try `rustc --explain E0384`.
error: could not compile `playground`.

{% include info.html text=“try rustc –explain E0384 to see what the error is all about” %}

This is freakin’ cool, now that variables cannot be reassigned, in a program, we do not have to worry if the types are going to change or if the state of a variable will change unintentionally.

Can a variable ever be a mutable?

In some situtations, we want variables to be reassigned like in a loop, or ehmm if we want to change our name to a new one.., for those situations, in Rust we can do:

fn main() {
    let mut name = "unrahul";    // just change the varible to a `mutable` variable
    println!("name is {}", name);
    name = "rahul";
    println!("new name is {}", name);
}

An the output is:

name is unrahul
new name is rahul

Shadowing

Shadowing in simple terms mean, a variable already declared can be redeclared with in a inner block with the same name.

For example, we can do this in Rust:

fn main() {
    let name = "unrahul";
    println!("name is {}", name);
    {
        let name = "rahul";    // here the variable is in inner scope (see the braces?), and this is shadowing the external declartion of `name`
        println!("new name is {}", name);
    }
}

Compiling and running this, we get:

name is unrahul
new name is rahul

Okay, I lied a bit, shadows need not be in inner or seperate block, this below is perfectly valid:

fn main() {
    let name = "unrahul";
    println!("name is {}", name);
    let name = "rahul";    // This is also considered as shadowing
    println!("new name is {}", name);
}

And the output of this is:

name is unrahul
new name is rahul

The end

Now that we have seen immutable and mutable variables, and also shadowing, may be primitive types next?..

Using cargo to create Rust projects

Cargo should be installed as part of rust installation, to ensure cargo is installed run

> cargo --version
cargo 1.41.0

you should be able to see something like above… to create a wireframe for a new project hworld run:

> cargo new hworld

This creates a directory hworld, with few files and one directory:

>
|-- Cargo.toml
|-- src
    |-- main.rs

Cargo goes one step ahead and adds versioning to the project, git is used by default, so if you do a ls -la you can see few more files, but for now let’s ignore that and focus on the main files.

• Cargo.toml - This is called the manifest file of our Rust project and is the file where we configure stuff like name, version of the project, Rust edition, the, author's, email address, and also there is a section to include the dependencies required by the project.

A sample Cargo.toml looks like:

[package]
name = "hworld"
version = "0.1.0"
authors = ["rahulunair <rahulunair@gmail.com>"]
edition = "2018"

[dependencies]
time = "0.1.12"

There are a lot more options that can be included in the manifest file, for an extensive list, see. Here I have specified one dependency time with version 0.1.12, we wont use it for anything now, but just for kicks eh?

• ./src/main.rs - It is a placeholder to add our code, cargo has populated it with:

fn main() {
    println!("Hello, world!");
}

Building the project with Cargo

It’s just as simple as running the below command from inside hworld directory:

> cargo build

If you are running it for the first time, you would see something like:

Updating crates.io index
  Downloaded time v0.1.42
  Downloaded libc v0.2.67
   Compiling libc v0.2.67
   Compiling time v0.1.42
   Compiling hworld v0.1.0 (hworld)
    Finished dev [unoptimized + debuginfo] target(s) in 49.31s

That’s it, we have built our first program using cargo.

Running the program

After cargo has built the project successfully, it creates a directory ./target/debug/ to save the binary, we can just cd to ./target/debug and run:

> ./hworld
Hello, world!

Yay! we have run our first Rust program, that was created using Cargo.

Releasing our code

Release version ideally should be an optimized binary, to obtain such a binary, it is as simple as running:

> cargo build --release

Running the above command on my system from hworld directory, Cargo prints:

   Compiling libc v0.2.67
   Compiling time v0.1.42
   Compiling hworld v0.1.0 (hworld)
    Finished release [optimized] target(s) in 8.42s

And, lo and behold there is an optimized binary hworld in hworld/target/release/, in later posts, we can see the difference between debug and release versions of binaries generated by Cargo.

The eligance of Cargo is that, everything is named as we would expect, no gotcha’s .. I really like that about Cargo.

The end

As a closing note, let’s recap Cargo mainly does two things for us,

• builds our project using - cargo build or cargo build –release
• builds and runs our project using - cargo run

Now, there is a third and final command I use, to check for compilation issues fast, without really building the binary, which can be slow at times.

The command to check if everything is alright is:

> cargo check

On my system it gives me this:

    Checking libc v0.2.67
    Checking time v0.1.42
    Checking hworld v0.1.0 (hworld)
    Finished dev [unoptimized + debuginfo] target(s) in 3.61s

That’s it, so we saw what cargo can do, it is a tool that can be used when we are building a Rust project, but if it is a simple file, with no dependencies we can still use rustc as we saw earlier in Flashcard Rust.

Setup

Hopefully, you are using Linux. To install Rust, use:

> curl https://sh.rustup.rs -sSf | sh

Rustup is a tool that is used to install Rust, the above script downloads and installs rustup first and then installs Rust.

Ahoy!

Let’s create a directory to keep our source files:

> mkdir ~/rust_sources
> cd ~/rust_sources

Our first program, to say ahoy!

> mkdir ahoy; cd ahoy
> touch main.rs

Open main.rs and add the following line:

fn main() {
    println!("ahoy ahoy!");  
}

Save the file and run rustc main.rs, thats it! We have compiled our first rust program.

There should be a file named main in the same directory main.rs is in, execute it using:

./main
ahoy ahoy!

Now that you have written 2 lines of rust, do read

The end

Magical number e

So let’s see how to find the value of e in three different ways. I think if you ask me what is your favorite number, I might say it is e, never ending, rather transcendental constant with value 2.7182818… . Funny thing is that this number comes up in many places across the universe, it was made famous by Bernoulli when he came across it, while studying compound interest..go figure out..>>

An example program to find value of e

There are lot of ways to find e, we can use the Binomial expansion, Newton’s method, Brother’s method (formualated in 2004 I think) etc.

package main

    import (
        "fmt"
        "math"
        "sync"
    )

    // dumbFact is a recursive function to print factorial
    func dumbFact(num float64) float64 {
        if num <= 1 {
            return 1
        }
        return num * dumbFact(num-1)
    }

    // Brother method to find e using approximation
    // for n=0 to limit
    //    (2n+2)/(2n+1)!
    func Brother(limit float64, wg *sync.WaitGroup) {
        var n float64
        var e float64
        // `defer` keyword will run the expression after the
        //function returns
        defer wg.Done()
        for n <= limit {
            e = e + (2*n+2)/dumbFact(2*n+1)
            n = n + 1
        }
        fmt.Printf("Brother's method for `e` yields: %20.15f\n", e)
    }

    // Binomial method to find `e`
    func Binomial(limit float64, wg *sync.WaitGroup) {
        defer wg.Done()
        e := math.Pow((1 + 1/limit), limit)
        fmt.Printf("Binomial solution to `e` is : %20.15f\n", e)
    }

    // Newton method - approximation when n is big
    func Newton(limit float64, wg *sync.WaitGroup) {
        defer wg.Done()
        var n float64
        var e float64
        for n <= limit {
            e = e + 1/dumbFact(n)
            n = n + 1
        }
        fmt.Printf("Newton's method gives the value as: %20.15f\n", e)
    }

    func main() {
        fmt.Println("A simple go routine example")
        // A waitgroup is just used to wait until the
        //routine completes
        var wg sync.WaitGroup
        wg.Add(3)
        go Brother(10000, &wg)
        go Binomial(10000, &wg)
        go Newton(10000, &wg)
        wg.Wait()
        fmt.Println("Done!")
    }

You can see the program in action here.

What’s happening here?

The juggling happens in lines 58, 59 and 60 of the program. Where, the scheduler will concurrently work on the three routines, there is nothing complicated in the program, it uses the basic factorial function, uses something called the WaitGroup and runs three go routines.

So why am I using a WaitGroup, you ask. Well, WaitGroup is just to prevent go from falling through and terminating the programming before the routines are done. It tells go to wait till all the routines have finished running. We could waited for a simple input from the user at the end after line 61 and made the program explicitly wait, which would have given the routines time to run as well. I arbitrarily chose this WaitGroup option.

The end

Essentially, go routines are simple at first, and that is important. As in any concurrent programming paradigms, there are complications and real world pains, but the entry level bar is low, which is in my opinion neat. This helps one learn the concepts of concurrency eventhough you might not use Go. Next time we shall see how to do something similar to this in python, or will we ?

What’s going on here?

All Singleton objects have the same share the same state - it is made possbile by the all powerful __dict__ attribute. It is a dictionary that contains all attributes and its values. We first assign __dict__ to refer to the dictionary refered to by _shared_data. This makes sure that every time an object for Singelton is created the __dict__ attribute of the Borg object is referring to the same instance variable _shared_data. Thus, when inheriting the class, the child class’s __dict__ is overwritten by parent class’s __dict__.

The end

As a closing note, about the philosphy of Borg, think about why we need a singleton? At least one of the reasons why we need one is to maintain a common state among different objects (some would say, that is the only reason). Borg does that for us, one state for all objects, for we are the Borg, we are all the same:)

The Borg pattern is so cool because it is so simple.