Lesson: Don’t worry about the schema.This fits with a piece I read the other day about how MongoDB has high adoption for small projects because it lets you just start storing things, without worrying about what the schema or indexes need to be. Reddit’s approach lets them easily add more data to existing objects, without the pain of schema updates or database pivots.
[Reddit] used to spend a lot of time worrying about the database, keeping everthing nice and normalized. You shouldn’t have to worry about the database. Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain. They would have to restart replication and could go a day without backups. Deployments are a pain because you have to orchestrate how new software and new database upgrades happen together.
Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.
The price is you can’t use cool relational features. There are no joins in the database and you must manually enforce consistency. No joins means it’s really easy to distribute data to different machines. You don’t have to worry about foreign keys are doing joins or how to split the data up. Worked out really well. Worries of using a relational database are a thing of the past.
Saturday, September 8, 2012
Reddit's database has only two tables
Saturday, September 1, 2012
What is sfsutl, How Should SFSUTL.EXE be Used?
in Siebel 7 and Siebel 8 a new utility called SFSCLEANUP has been introduced.
sfsutl can be used to verify the integrity between database records and files in the Siebel File System, or to remove files from the Siebel File System that no longer have a corresponding database record.
SFSUTL works in two modes:
1. verify or report
2.move or delete.
1.Verify or Report Mode:
To use the utility in verify or report mode, enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /x c:\sfsutl_output.txt
The output from this command could be easily imported into Microsoft utilities like Excel or Access for sorting and manipulation. See below for sample output from this command. There are three key statuses to note:
•Keeping - The file corresponds to a database record.
•Discard - The file does not correspond to a database record, and would be moved or deleted if sfsutl were run in that mode.
•Missing - An expected file does not exist for a database record.
2.Move or Delete Mode:
To use the utility in move or delete mode:
1.Create a separate or new folder (this will be used by sfsutl when moving the discarded files). It is recommended that users do not create this new folder as a subfolder of the existing Siebel File System. For example, if the Siebel File System is in e:\siebfile then create the new folder as e:\siebfilediscards.
2.Enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /m e:\siebfilediscards /x c:\sfsutl_output.txt
Wednesday, August 15, 2012
Siebel Business Services
- Asynchronous Server Requests
- Data Validation Manager
- EAI Data Transformation Engine
- EAI Dispatch Service
- EAI File Transport
- EAI JMS Transport
- EAI Siebel Adapter
- EAI XML Converter
- EAI XML Queuing Service
- EAI XML Write to File
- EAI XML Read from File
- EAI XSLT Service
- FINS Data Transfer Utilities
- FINS Teller Converter Extensions
- FINS Teller UI Navigation
- Outbound Communications Manager
- PRM ANI Utility Service
- Read CSV File
- Row Set Transformation Toolkit
- SIA BC Utility Service
- SIS OM PC Service
- SIS OM PMT Service
- Shopping Service
- Spell Checker
- SSSE Address Parser (eScript)
- Synchronous Server Requests
- TNT SHM Workflow UI Service
- Transcode Service
- Unified Messaging Service
- Universal Inbox
- Usage Tracking Service
- Web Engine HTTP TXN
- Workflow Process Manager
- Workflow Utilities
- XML Gateway
Saturday, July 7, 2012
Tools of the trade
Siebel Development
- A good text editor is essential for any development need. For years I have been relying on window’s default notepad utility, which is good enough. But it is seriously lacking in features, and hangs when you try to open huge files. I use Notepad2 , Notepad++ and EditPlus. Each has first class CR/LF support, ANSI to Unicode switching, whitespace and line ending graphics and Mouse Wheel Zooming. EditPlus even has ability search and replace special characters (\n,\t) and works nicely with huuge files.
- If you work deals with XMLs/XSLs, you will need a good XML/XSLT/WSDL editor. XmlSpy has been my number one choice, followed by Oxygen. Though I have noticed that there are some minor differences in the XMLSpy XPATH engine between older and newer versions. The XPATH engine in Spy version 6 is similar to Siebel’s own EAI XSLT service. Some of the fine working XSLTs were throwing validation errors in the newer version of Spy, but they worked perfect in Siebel.
- SOAP UI. This one deserves a special mention. Not only is it free, it has support for secure webservices , proxies and many other protocols. Its so easy to set up a stub/mockservices, and these have saved my life more than once.
- Fiddler. I first used this way back in college, when I wanted to intercept and see the http messages being sent from my system for a project. It’s a free utility from Microsoft, and comes in handy in diagnosing web service related issues. Its got a scripting system, and can be used as mockservice too.
- UnixUtils. A unix developer swears by the huge number of commands at his disposal. Sadly, windows does not provide that many commands, and I thank the geeks for porting these awesome Unix commands over to Win32. These allow me to build bat files and scripts to solve recurring issues.
- Although UnixUtils has windows port of grep and tail commands, the UI of BareGrep and BareTail and much more developer friendly.
- Oracle SQLDeveloper, TOra and my old favorite, TOAD. You can’t be a Siebel developer without knowing SQL !
- I’m sure all you Siebel developers have been in this situation: You have list of data you need to query in Siebel :
val1
val2
val3
The searchspec you need to build is : “val OR val2 OR val3” . Well, that’s all that Orit does. Frankly, I don’t know who developed this utility, but it is something developed by a Siebel developer. All it does is simply concatenate column wise data with ‘OR’ in between.
Other Siebel Things I found on the net
- The good folks at the Only Siebel blog have released Excel Macro tools which can pull config data for documentation.
- Oli has graciously released a lot of his tools for free download. There are script analysers, data loaders and Tracer Tools.
- Some useful utilities at the website of Sea Marvel Tech Solutions
- Wait…one more utility for Siebel Documentation here.
- Here is my own attempt at making an SQL Tracer.
- Siebel provides dbisqlc to connect and run SQL statements on your localdb. But I find it painful when there are too many columns in the output, I have to scroll to see the data, and export. Sel2XL is a nice excel macro which does the same thing, and it allows excel formatting.
- Sometimes, the easiest way out is direct SQL into the the database. There are SQLs for siebel issues, EIM Mappings, for UI mappings, and so many other issues.
General Use
- Window’s default search feature can get irritatingly slow, specially if you need to search across drives and have to run complex search patters. Everything is an awesome ,high speed ,instant search utility for the NTFS file system. You have to try it to see the magic. And did I mention it is free ?
- Teracopy is the default copy utility on my systems. Sure, the latest multicopy feature on Windows 7 & 8 are cool, but I still feel Teracopy’s interface is better.
- Free Download Manager. For all those loong downloads.
- Microsft’s Virtual PC. I use this to setup virtual machines for my Siebel POCs. Its free and no fuss to setup and use. This new version of Windows Virtual PC lets you run Windows XP applications next to your Windows 7 apps for the ultimate in backward-compatibility.
- I have been using Liberkey suite since they came out some years ago. They have a free utility for all your day to day needs. NirSoft Utilities Collection is also a good choice.
- Autohotkey, DoItAgain. Ultimate Windows Automation.
Wednesday, April 18, 2012
View Layout Cache in Siebel Enterprise
View Layout Cache in Siebel Enterprise:
This is a subject that comes up often enough in the classroom or in the office, so I thought we would post a short article about it. Before we begin, we need to set the scene. You are navigating through Siebel Call Center or whatever application you use, and you click Help - About View with the Shift key held down. Siebel will kindly tell you how the View Layout you were looking at was brought to you :
Not Cached :
Well, bad news. You are on a dynamic layout view (amongst other things, applets such as Explorer, Hierarchical, Dynamic Drilldowns, Personalization rules can all make the View Layout so dynamic as to be ”uncacheable”). So the View Layout is never cached.
Server :
Well, so we are getting somewhere. The View Layout was retrieved from the Server and Web Server Cache.
Memory :
Potentially even better. The View Layout came from your Browser Cache. In fact, maybe you just got “Server” as the response, then returned to the same View a few seconds later. The Layout was now in your Browser cache, so you got it from the Browser not your Server. Cool!
Disk :
Your Siebel Object Manager on your Siebel Server has the WebTemplateVersion parameter set and there was no updated View Layout available, so the Persistent Disk Cache version was used. The View Layout you requested was found in the Browser Persistent Disk Cache. This may even have happened between sessions in the Application. Cool!
Digging around in your Browser cache folder will bring all sorts of interesting surprises if this is set up. For example, here is a cached HTML file that was in my Browser just after the previous session :
I guess the point here is to think about the different situations where this may impact us.
- Testing Average Load Times
- Testing Usage Scenarios and making Assumptions
- Choosing View Layout Templates carefully
- Putting Explorer or other dynamic Applets in Views
- Clearing your Browser Cache
Sunday, March 11, 2012
Siebel 8 Script Libraries
Quick question; will the following code snippet work ?
Business Service : BS1, contains only this code
function function1 ()
{
TheApplication().RaiseErrorText("function1 triggered");
}
There is no code in any other event/function of this BS. And now, the attempt is to trigger this BS via the following code:
var bs = TheApplication().GetService("BS1");
bs.function1();
Now there is something wrong about the second code snippet, right ? This is not the usual way to invoke a Business Service Method. The practise is to use the InvokeMethod command, passing property sets for input and output. But here is the output of running these in Siebel 8
This is an example of Script Libraries feature from Siebel 8 onwards. Developers can write multiple functions in business services, and then these functions get exposed , and the functions can be invoked directly as you would do on C/C++/Java. There is no need of adding code in Service_PreInvokeMethod event to expose the functions.
There are limitations though, such a business service’s functions can be invoked only via scripting. They cannot be used in WFs or BRPs. But if your functionality calls for lots of scripting, this feature surely comes in handy.
The ever friendly Oli has been posting some really tricky pieces of code for his code challenges. Head over there to learn scripting mistakes that creep up in code.
Happy Scripting !
Friday, March 9, 2012
View Refresh when clicking New Record in a view with Dynamic Toggle
Dynamic toggle applets were probably the first piece of automation a Siebel developer gets to work on; switch the applet depending on some field value. No scripting, nothing at the BusComp level, just Applet toggles. But issues crop in when the logged in user tries to create new record on the applet. Sometimes the view jumps or refreshes, specially when there are lots of applets stacked in the view. The user has to manually scroll down back to this target applet.
Oracle has a work around for this ‘defect’ documented here [ID 541100.1] , which involves loads of scripting at the applet and BusComp level. But I tried to come up with something with fewer lines of code.
Resulting solution: add the following code in the WebApplet_PreInvokeMethod section of the base applet as well as its toggle applets:
if (MethodName == "NewRecord")
{
this.BusComp().NewRecord(NewBefore);
return (CancelOperation);
}
Yep, I know, the code does not make sense at all. But for some reason, it works ! The view does not jump and the new record gets created right there in the applet. At at just 3 lines of code, it beats oracles long and elaborate code version.