Saturday, September 8, 2012

Reddit's database has only two tables

Steve Huffman talks about Reddit’s approach to data storage in a High Scalability post from 2010. I was surprised to learn that they only have two tables in their database.
Lesson: Don’t worry about the schema.
[Reddit] used to spend a lot of time worrying about the database, keeping everthing nice and normalized. You shouldn’t have to worry about the database. Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain. They would have to restart replication and could go a day without backups. Deployments are a pain because you have to orchestrate how new software and new database upgrades happen together.
Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.
The price is you can’t use cool relational features. There are no joins in the database and you must manually enforce consistency. No joins means it’s really easy to distribute data to different machines. You don’t have to worry about foreign keys are doing joins or how to split the data up. Worked out really well. Worries of using a relational database are a thing of the past.
This fits with a piece I read the other day about how MongoDB has high adoption for small projects because it lets you just start storing things, without worrying about what the schema or indexes need to be. Reddit’s approach lets them easily add more data to existing objects, without the pain of schema updates or database pivots.
.

Saturday, September 1, 2012

What is sfsutl, How Should SFSUTL.EXE be Used?


sfsutl.exe can be found in Siebsrvr\bin directory, in siebel versions 6 and older SFSUTL.EXE used to delte the orphan and old versions of attachments.

in Siebel 7 and Siebel 8 a new utility called SFSCLEANUP has been introduced.
sfsutl can be used to verify the integrity between database records and files in the Siebel File System, or to remove files from the Siebel File System that no longer have a corresponding database record.
SFSUTL works in two modes:

1. verify or report

2.move or delete.
1.Verify or Report Mode:
To use the utility in verify or report mode, enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /x c:\sfsutl_output.txt
The output from this command could be easily imported into Microsoft utilities like Excel or Access for sorting and manipulation. See below for sample output from this command. There are three key statuses to note:
•Keeping - The file corresponds to a database record.
•Discard - The file does not correspond to a database record, and would be moved or deleted if sfsutl were run in that mode.
•Missing - An expected file does not exist for a database record.
2.Move or Delete Mode:
To use the utility in move or delete mode:
1.Create a separate or new folder (this will be used by sfsutl when moving the discarded files). It is recommended that users do not create this new folder as a subfolder of the existing Siebel File System. For example, if the Siebel File System is in e:\siebfile then create the new folder as e:\siebfilediscards.
2.Enter a command string similar to the following:
sfsutl /u sadmin /p sadmin /c siebsrvr_siebel /d dbo /f e:\siebfile /m e:\siebfilediscards /x c:\sfsutl_output.txt

Saturday, July 7, 2012

Tools of the trade


Five years of Siebel EAI development, and I am still learning every day. One thing I quickly realized was that the Siebel Tools and Application system is not like the contemporary Integrated Development Environment (IDE) used in Java and .NET development. Sure, Tools can solve most of your problems, but every Siebel developer will eventually face a problem which has to be solved using some other tool/utility/script which may have to be written from scratch. Today, I thought I’d share what’s in my arsenal of tools.

Siebel Development

  1. A good text editor is essential for any development need. For years I have been relying on window’s default notepad utility, which is good enough. But it is seriously lacking in features, and hangs when you try to open huge files. I use Notepad2 , Notepad++ and EditPlus. Each has first class CR/LF support, ANSI to Unicode switching, whitespace and line ending graphics and Mouse Wheel Zooming. EditPlus even has ability search and replace special characters (\n,\t) and works nicely with huuge files.
  2. If you work deals with XMLs/XSLs, you will need a good XML/XSLT/WSDL editor. XmlSpy  has been my number one choice, followed by Oxygen. Though I have noticed that there are some minor differences in the XMLSpy XPATH engine between older and newer versions. The XPATH engine in Spy version 6 is similar to Siebel’s own EAI XSLT service. Some of the fine working XSLTs were throwing validation errors in the newer version of Spy, but they worked perfect in Siebel.
  3. SOAP UI. This one deserves a special mention. Not only is it free, it has support for secure webservices , proxies and many other protocols. Its so easy to set up a stub/mockservices, and these have saved my life more than once.
  4. Fiddler. I first used this way back in college, when I wanted to intercept and see the http messages being sent from my system for a project. It’s a free utility from Microsoft, and comes in handy in diagnosing web service related issues. Its got a scripting system, and can be used as mockservice too.
  5. Network tools like PuTTy, WinSCP and FileZilla come here. I also use PuTTY to setup ssh tunnels when I want to route a request via another server.
  6. UnixUtils. A unix developer swears by the huge number of commands at his disposal. Sadly, windows does not provide that many commands, and I thank the geeks for porting these awesome Unix commands over to Win32. These allow me to build bat files and scripts to solve recurring issues.
  7. Although UnixUtils has windows port of grep and tail commands, the UI of BareGrep and BareTail and much more developer friendly.
  8. Oracle SQLDeveloper, TOra and my old favorite, TOAD. You can’t be a Siebel developer without knowing SQL !
  9. I’m sure all you Siebel developers have been in this situation: You have list of data you need to query in Siebel :
val1
val2
val3

The searchspec you need to build is : “val OR val2 OR val3” . Well, that’s all that Orit does. Frankly, I don’t know who developed this utility, but it is something developed by a Siebel developer. All it does is simply concatenate column wise data with ‘OR’ in between.

Other Siebel Things I found on the net

Its sad that Siebel does not provide an Out-of-box documentation feature, which can be used during and development. This is just one of the many problems I have faced.Nice thing is others have faced the same problem, and developed their own solutions for the problem.
  1. The good folks at the Only Siebel blog have released Excel Macro tools which can pull config data for documentation.
  2. Oli has graciously released a lot of his tools for free download. There are script analysers, data loaders and Tracer Tools.
  3. Some useful utilities at the website of Sea Marvel Tech Solutions
  4. Wait…one more utility for Siebel Documentation here.
  5. Here is my own attempt at making an SQL Tracer.
  6. Siebel provides dbisqlc to connect and run SQL statements on your localdb. But I find it painful when there are too many columns in the output, I have to scroll to see the data, and export. Sel2XL is a nice excel macro which does the same thing, and it allows excel formatting.
  7. Sometimes, the easiest way out is direct SQL into the the database. There are SQLs for siebel issues, EIM Mappings, for UI mappings, and so many other issues.

General Use

  1. Window’s default search feature can get irritatingly slow, specially if you need to search across drives and have to run complex search patters. Everything is an awesome ,high speed ,instant search utility for the NTFS file system. You have to try it to see the magic. And did I mention it is free ?
  2. Teracopy is the default copy utility on my systems. Sure, the latest multicopy feature on Windows 7 & 8 are cool, but I still feel Teracopy’s interface is better.
  3. Free Download Manager. For all those loong downloads.
  4. Microsft’s Virtual PC. I use this to setup virtual machines for my Siebel POCs. Its free and no fuss to setup and use. This new version of Windows Virtual PC lets you run Windows XP applications next to your Windows 7 apps for the ultimate in backward-compatibility.
  5. I have been using Liberkey suite since they came out some years ago. They have a free utility for all your day to day needs. NirSoft Utilities Collection is also a good choice.
  6. Autohotkey, DoItAgain. Ultimate Windows Automation.

Wednesday, April 18, 2012

View Layout Cache in Siebel Enterprise

This is reblogged from http://ondemand-education.com. Those guys have some cool articles :-)


View Layout Cache in Siebel Enterprise:
This is a subject that comes up often enough in the classroom or in the office, so I thought we would post a short article about it. Before we begin, we need to set the scene. You are navigating through Siebel Call Center or whatever application you use, and you click Help - About View with the Shift key held down. Siebel will kindly tell you how the View Layout you were looking at was brought to you :
Not Cached :
Siebel View Layout Not Cached
Well, bad news. You are on a dynamic layout view (amongst other things, applets such as Explorer, Hierarchical, Dynamic Drilldowns, Personalization rules can all make the View Layout so dynamic as to be  ”uncacheable”). So the View Layout is never cached.
Server :
Server Cache  View Layout
Well, so we are getting somewhere. The View Layout was retrieved from the Server and Web Server Cache.
Memory :
Memory Cached View Layout
Potentially even better. The View Layout came from your Browser Cache. In fact, maybe you just got “Server” as the response, then returned to the same View a few seconds later. The Layout was now in your Browser cache, so you got it from the Browser not your Server. Cool!
Disk :
Cached View Layout from Disk
Your Siebel Object Manager on your Siebel Server has the WebTemplateVersion parameter set and there was no updated View Layout available, so the Persistent Disk Cache version was used. The View Layout you requested was found in the Browser Persistent Disk Cache. This may even have happened between sessions in the Application. Cool!
Digging around in your Browser cache folder will bring all sorts of interesting surprises if this is set up. For example, here is a cached HTML file that was in my Browser just after the previous session :
Cache Content
I guess the point here is to think about the different situations where this may impact us.
  • Testing Average Load Times
  • Testing Usage Scenarios and making Assumptions
  • Choosing View Layout Templates carefully
  • Putting Explorer or other dynamic Applets in Views
  • Clearing your Browser Cache
There are a number of parameters  both in the CFG side of things and in the User Preferences that can affect the overall behavior, so watch out and take note of the Bookshelf on the subject of Improving Performance.

Sunday, March 11, 2012

Siebel 8 Script Libraries

 

Quick question; will the following code snippet work ?

Business Service : BS1, contains only this code

function function1 ()
{
TheApplication().RaiseErrorText("function1  triggered");
}

There is no code in any other event/function of this BS. And now, the attempt is to trigger this BS via the following code:

var bs = TheApplication().GetService("BS1");
bs.function1();

Now there is something wrong about the second code snippet, right ? This is not the usual way to invoke a Business Service Method.  The practise is to use the InvokeMethod command, passing property sets for input and output.  But here is the output of running these in Siebel 8

image

 

This is an example of Script Libraries feature from Siebel 8 onwards. Developers can write multiple functions in business services, and then these functions get exposed , and the functions can be invoked directly as you would do on C/C++/Java. There is no need of adding code in  Service_PreInvokeMethod  event to expose the functions.

There are limitations though, such a business service’s functions can be invoked only via scripting. They cannot be used in WFs or BRPs. But if your functionality calls for lots of scripting, this feature surely comes in handy.

 

The ever friendly Oli has been posting some really tricky pieces of code for his code challenges. Head over there to learn scripting mistakes that creep up in code.

Happy Scripting !

Friday, March 9, 2012

View Refresh when clicking New Record in a view with Dynamic Toggle

 

Dynamic toggle applets were probably the first piece of automation a Siebel developer gets to work on; switch the applet depending on some field value. No scripting, nothing at the BusComp level, just Applet toggles. But issues crop in when the logged in user tries to create new record on the applet. Sometimes the view jumps or refreshes, specially when there are lots of applets stacked in the view. The user has to manually scroll down back to this target applet.

Oracle has a work around for this ‘defect’ documented here [ID 541100.1] , which involves loads of scripting at the applet and BusComp level. But I tried to come up with something with fewer lines of code.

Resulting solution: add the following code in the WebApplet_PreInvokeMethod section of the base applet as well as its toggle applets:

 

if (MethodName == "NewRecord")
{
this.BusComp().NewRecord(NewBefore);
return (CancelOperation);
}

 

Yep, I know, the code does not make sense at all. But for some reason, it works ! The view does not jump and the new record gets created right there in the applet. At at just 3 lines of code, it beats oracles long and elaborate code version.