Joining the PRIOR team at AllenAI this summer.ġ paper accepted to NeurIPS 2023 on improving the reasoning abilities of open multimodal LLMs with question decomposition. Two papers accepted to CVPR 2024, on self-training agents to solve computer vision tasks via program synthesis (summer internship work with NEC Laboratories) and black-box predictive uncertainty for multimodal LLMs. Outside of research, I lift weights, read (here’s my goodreads profile), watch mixed martial arts, and sometimes wonder whether randomness is real.īecoming a member of Mohit Bansal’s group ( MURGe-Lab) at UNC Chapel Hill as a PhD student, where I’ll be working on multimodal agents, grounded language reasoning, and other exciting vision/language topics! Before graduate school, I spent ~3 years as an early member of the engineering / data science organizations at two high growth startups: Roadie ( acquired by UPS for $500m) and Intelligent Flying Machines / OneTrack.AI as software engineer, where I led efforts to scale data infrastructure to match growth, and worked on a range of challenging problems, including embedded deep learning, fault-tolerant distributed systems, realtime adaptive pricing, and data pipelines. I completed my Masters in Computer Vision and Learning Algorithms at Northeastern University in Boston under Raymond Fu in close collaboration with the Media Analytics Group at NEC Laboratories America under Manmohan Chandraker, where I worked on grounded language understanding and reasoning.ĭuring my Masters, I won an university-wide Outstanding Graduate Student award for my work on the (mis) use of racial categories in computer vision ( Scroll.IN reporting, reporting).

A problem I’ve been thinking about is selective prediction for open-ended visual question answering, because the uncertainty can come from both the language model itself, as well as the binding between vision and language. A requirement for high-stakes applications (and even personal use) for any AI system is an ability to say “I don’t know”. Uncertainty quantification and reliable models.Neurosymbolic systems in general, but especially approaches using program synthesis as a way to represent reasoning formally, disentangle reasoning from perception and impose constraints on behavior.Can we improve the LLM from interactive feedback? This has been done in formal environments like LeanDojo for theorem-proving, but how do we construct virtual environments with feedback for tasks like open-world understanding? Systems like Voyager, ViperGPT use LLMs as planners, but keep them frozen. Foundation model-driven agents that learn from grounded interaction.For example, GPT-4V is still far from solving WinoGround, and EncylopedicVQA is difficult even for PaLI. Grounded, complex reasoning tasks, such as interactive theorem proving or open-world image understanding.I completed my BS+MS at Northeastern, where I was fortunate to be advised by Raymond Fu. I’m also a student researcher in the Media Analytics Group at NEC Laboratories America with Manmohan Chandraker (from 2022). I’m an incoming PhD student at Mohit Bansal’s’s group ( MURGe Lab) at UNC Chapel Hill. It’s odd, considering that most users are likely using the GGUF quant as opposed to EXL2.My goal is to build trustworthy, teachable multimodal language-driven agents that can reason and write code. The GGUF quant is still afflicted with identical settings. Penalty at 1.0.Įdit: If anyone who has had success with Yi-34B models could kindly list what quant, parameters, and context they’re using, that may be a good start for troubleshooting.Įdit 2: After trying various sampling parameters, I was able to steer the EXL2 quant away from repetition - however, I can’t speak to whether this holds up in higher contexts. The models appear to be very promising from Wolfram’s evaluation, so I’m wondering what error I could be making.Ĭurrently using Text Generation Web UI with SillyTavern as a front-end, Mirostat at Tau values between 2~5, or Midnight Enigma with Rep.

I was wondering if anyone else has experienced similar issues with the latest finetunes, and if they were able to resolve the issue. This appears to persist with both GGUF and EXL2 quants, and happens regardless of Sampling Parameters or Mirostat Tau settings. Messing around with Yi-34B based models (Nous-Capyabara, Dolphin 2.2) lately, I’ve been experiencing repetition in model output, where sections of previous outputs are included in later generations.

Finetunes distribution